Finalizing notes and team debriefings
Researchers review the collected data to ensure that the notes and descriptions are written clearly such that someone who did not participate in the data collection can easily understand what happened or what was said. Researchers also organize their notes so that other researchers can easily identify where topics were discussed or observed.
If multiple researchers conducted the data collection activity together, each researcher’s notes are reviewed by the other researcher(s), so that the other researcher(s) can add any additional information to ensure the comprehensiveness of the data.
Often, research projects require site visits to multiple locations or programs to conduct interviews, focus groups, and/or observations. For projects of this nature, it is usually not feasible for just one researcher or pair of researchers to conduct all of the qualitative data collection. When multiple researchers are conducting this work, they meet periodically throughout the data collection period to have team debriefing meetings. During these meetings, researchers report their preliminary impressions from data collection to date. The meetings allow researchers to identify issues that are recurring across different sites or aspects of a site that are unique. Researchers also discuss the strengths and weaknesses of the different types of evidence and possible alternative explanations (Martinson and O’Brien 2010).
Coding
Coding is one method for creating analytic files and documenting and validating data across all members of the research team. It is a process of assigning codes, words, or phrases that identify to which topics or issues portions of the data refer, and organizing the data in a way that is useful for further analysis (Bailey 2007). There are several steps to coding, which is often an iterative process.
1. Developing a preliminary coding scheme. A coding scheme is a set of codes, defined by the words and phrases that researchers assign to categorize a segment of the data by topic. To develop a preliminary coding scheme, researchers consider what questions they are trying to answer and the related topics to those questions. Discussions in debriefing meetings also help identify recurring or key issues and can inform what codes are necessary. At this stage, a coding dictionary is also developed.
2. Testing and refining the coding scheme. Once a preliminary coding scheme has been developed, a small group of researchers review and code a small subset of the qualitative data collected using the preliminary scheme. After coding, the researchers meet to discuss how they coded the data. This process allows researchers to identify issues that are raised in the data for which there is not yet a code. They also discuss each code to refine the definitions so that each code is clearly operationalized and mutually exclusive from other codes.
3. Testing inter-coder reliability. If multiple researchers are responsible for coding the data, researchers conduct inter-coder reliability tests. Using the refined coding scheme, researchers responsible for coding the data select another small subset of the data to code. After coding is completed, they examine the extent to which researchers are coding the data in the same way. By using a computer assisted qualitative data analysis software package, the research team can run tests to calculate the kappa coefficient, a statistical measure that identifies the probability that data were coded in the same way across researchers by chance, for each code. This identifies codes that researchers are not applying consistently. Researchers meet again to discuss the subset of inconsistent codes to further refine their definitions. This process can be completed as many times as is necessary for the research team to feel confident that the coders are applying the coding scheme consistently.
4. Coding. Coding can be done by hand, entering data into a spreadsheet or database, or can be done using computer assisted qualitative data analysis software. The latter is useful for synthesizing large amounts of qualitative data.
5. Reviewing data by code. Once all the data are coded, researchers can sort the data by code. This is a useful way of sorting information on a single topic across respondents and/or sites and can be used to identify patterns, trends, and themes.
Creating summary tables and charts
It can also be useful to tabulate the number of sites or respondents that meet a certain condition to better understand how widespread that condition is or map which conditions are met where. For example, to understand collaboration between organizations for a certain project, researchers might identify strategies through data review and coding. They could then tabulate the number of sites that mentioned using the identified strategies, such as co-location, joint case management plans, joint team meetings, or data sharing agreements. This method identifies the most commonly used strategies, combinations of methods, and patterns across sites with respect to the strategies used.
Analyzing data by themes across all cases or sites can also be used to examine how data collection and analysis findings compare with expectations and hypotheses (Yin 2003).
Other data sorting and visualization strategies
Sorting and arraying data in many different ways can expose or create new insights and can identify conflicting data that may disconfirm the analysis (Martinson and O’Brien 2010). Studies of program development and implementation can benefit from detailed timelines for each site from program design and development through implementation. Researchers can also prepare detailed diagrams of key elements of program design and operation to compare sites and to assess variations between program design and on-the-ground implementation. Another strategy involves mapping participant flow through the various program elements.
Drafting case studies or case memos
Qualitative data can also be used to assemble a detailed description of one place, program, or group. These nuanced descriptions are commonly referred to as case studies or case memos. For example, a study’s research questions might require the researchers to document how a program was implemented and the challenges the program staff faced during this phase, for each of the sites selected for the sample. To facilitate answering this question, researchers can write a detailed case study for each site where data collection occurred. These case studies can then be compared to identify similarities and differences across sites and recognize emerging trends.
Throughout the analysis process, the researcher must remain open to new interpretations and insights. The data analysis ends when the best possible fit has been reached between the observations and interpretations (GAO 1991).
Related research
New Markets Tax Credit (NMTC) Program Evaluation: Final Report
A Descriptive Study of Tribal Temporary Assistance for Needy Families (TANF) Programs
References
Bailey, Carol A. 2007. A Guide to Qualitative Field Research. 2nd ed. Thousand Oaks, CA: Pine Forge Press, Sage Publications.
GAO (US General Accounting Office). 1991. Case Study Evaluation. Washington, DC: GAO.
Huberman, A. Michael, and Matthew B. Miles. 1994. “Data Management and Analysis Methods.” In Handbook of Qualitative Research, edited by Norman K. Denzin and Yvonna S. Lincoln, 428–44.Thousand Oaks, CA: Sage Publications.
Martinson, Karin, and Carolyn O’Brien. 2010. “Conducting Case Studies.” In Handbook of Practical Program Evaluation, 3rd ed., edited by Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer. San Francisco: John Wiley & Sons.
Yin, Robert K. 2003. Case Study Research: Design and Methods. 3rd ed. Thousand Oaks, CA: Sage Publications.