literature review challenges: search terms and depth

As I build my preliminary bibliography for the games & tourism literature review, I’m running into a number of challenges.

First, as I mentioned in my workflow post, the ubiquity of the word “game” makes it difficult to construct a search that yields the right results. A search that returns 70,000 results is almost as useless as one that yields only two or three. What has been more productive has been identifying an initial set of strong articles (both theoretical and case study) from both computing and tourism literatures, and then reviewing the bibliographies of those articles as well as the list of citing articles (via Google Scholar searches). I began with about 150 articles that I’d tentatively identified as relevant, and have been working my way through them, alphabetically by title, moving the useful items to a “vetted” collection, and adding the new items from bibliographies/citations as I go. I’m nearly through the articles beginning with G, and now have over 300 articles in the preliminary collection.

This leads to the next problem, which is knowing how far I should follow down the rabbit hole of prior work. For instance, if a case study on a tourism game cites an article on co-creation in tourism, and that article in turns cites an earlier theoretical work, do I need to acknowledge and include all three articles? This is a little easier for me to answer in the computing and game studies literature, since I have a better sense of what needs to be explicitly acknowledged/summarized, and what can be implied. But I don’t have that same understanding of the tourism literature, especially when it comes to core concepts like experience design and co-creation.

For now, I’m erring on the side of inclusion, since it’s easier to remove or ignore the articles when I start writing than it would be to track them back down again. But it’s making the preliminary evaluation of citations a LOT slower than I’d initially hoped.

literature review tools & workflow

Since a number of people asked me on Facebook about my literature review process, I decided to write up a detailed version of my answer.

First, the tools I’m using. On my Mac: Zotero citation manager (stand-alone software plus Chrome browser extension), university VPN for access to my university library’s database subscriptions. And on my iPad: PaperShip for access to my Zotero library, VPN for library database access, Zotero bookmarklet for mobile Safari.

(All of those tools are free, although I paid $10 to upgrade PaperShip so that I could annotate PDFs inside the app, and I pay $20/year for 2GB of Zotero cloud storage so that I have access to the PDFs of articles from any of my devices.)

On to the process…

I started with a series of searches in Google Scholar for terms related to my research (the intersection of games and tourism). This turned out to be particularly challenging (even for a trained librarian like me), since the word “game” can be used in so many ways–i.e. olympic games, game animals, game theory. As I encountered citations that looked like they’d be relevant, I used the Zotero extension in Chrome to add those items to a collection I called “Preliminary Lit Review.” In most cases, I made decisions based on title and abstract, since this was a first pass. (I did exclude works that were from known predatory publishers, like igi-global.) When I’m logged into our university’s VPN, the plugin does an excellent job of retrieving the full text of the article along with the citation. In the few cases where it didn’t, I attempted to find the PDF myself using our library’s online databases, and then ordered the remaining items via interlibrary loan.

When I found articles that looked particularly relevant, I used Google Scholar’s “cited by” feature to find related work. Once I started running out of relevant works in Scholar, I re-ran my searches in a number of databases that our university subscribes to, including ACM DL and IEEExplore for computing literature, and SpringerLink, Web of Science, and ProQuest for tourism literature. For each of these searches, I again used the Zotero plugin to add the citations. At the end of this process, I had about 180 citations, complete with full text of the item, stored in my “preliminary” collection.

I then began working my way through the preliminary collection. I open the full text of the article, and scan through it to see if it has clear relevance to the review I’m doing. I’m excluding games that are entirely virtual/digital, with no connection to real world exploration. I’m also excluding items that mention games so peripherally that they add no additional information to the review. For each of these items, I’m adding a note to the Zotero entry so that I can remember later why it wasn’t included.

If the article seems relevant, I then add tags to help me categorize it for later steps. While I’ll need to add more tags once I start writing and develop a structure for organizing the literature, right now I’m primarily focused on simple descriptors like “case study”, “mobile game”, and “augmented reality”. I also review the article’s bibliography to see if there are additional citations that look useful. For articles that seem particularly useful, I’ll check Google Scholar so that I can look over the “cited by” data. I then copy the citation to the “final” review collection.

Working my way through the preliminary collection in this way, I seem to have gotten close to a saturation point at only about the 20% mark. The preliminary collection has grown to ~240 citations, but increasingly the bibliographies and referencing works are items that I already have in the preliminary collection. I suspect there won’t be a lot more that need to be added.

I’m using my iPad when I’m away from my apartment. PaperShip gives me access to my Zotero library, and I can download any of the full text papers from my Zotero cloud storage. PaperShip also provides annotation tools that I can use to either add notes using the iPad keyboard, or directly on the page using the Apple Pencil. Annotations are synced with the PDF, and available on my computer later. This makes it easy for me to do the more intensive reading wherever I am, without carrying the computer with me (something that’s more of a challenge as I recover from my broken foot).

I’m also using Microsoft Word to take notes as I work; I save the file to Dropbox, and can then access it with Word on my iPad as well. I didn’t list Word in the tools above, since notetaking can be done with any editor.