The Potential Methodological Innovation of Qualitative Comparative Analysis in Applied Linguistics And TESOL

Yu Tang, University of Vienna, Vienna, Austria 
Andy Curtis, City University of Macau, Macau SAR, China

Introduction

In recent years, methodological issues in applied linguistics (AL) have received significant attention, as evidenced by leading journals (e.g., Applied Linguistics, Language Learning, and TESOL Quarterly) highlighting the need for methodological innovation (Phakiti et al., 2018). For example, even though the field of AL has been around for a century or more, just two years ago, in 2022, a new journal, Research Methods in Applied Linguistics (RMLA), was launched. It is rare that new journals are launched in our field, and such a launch reaffirms the acknowledgment of an unmet need, in this case, the need for methodological innovation in AL. According to the Aims and Scope of RMLA, applied linguistics is “a discipline that explores real-world language-related issues and phenomena.” Leaving aside the question of what an “un-real world” would look like, the co-editors also state that their journal “encompasses all aspects of research methods [and] welcomes research from all paradigms, be they quantitative, qualitative, or mixed, and methods of all kinds, whether they are utilized to observe the occurrence of a phenomenon or behavior, explore correlations, or examine causal relationships” (emphasis added). It is the examination of those causal relationships in our work as TESOL professionals and applied linguists where Qualitative Comparative Analysis (QCA) may be of most use.

In the majority of AL research, understanding the drivers behind educational, social, political, and economic outcomes is crucial for policymakers, academics, and practitioners alike. However, existing research methods in AL, as the launch of the new AL journal indicates, can fall short of capturing the complexity of influencing factors and the intricate causal configurations leading to outcome variables. This is where QCA could help. Originally pioneered by University of California professor Charles Ragin in the late 1980s and updated in the mid-2010s (Ragin, 1987, 2014), Ragin (2014) stated that, “The most distinctive aspect of comparative social science is the wide gulf between qualitative and quantitative work” (p. 2). And while we might like to believe that that gulf has been bridged, in reality, those of us who work with words and who gather primarily qualitative data, are still finding that numbers and numerical data often still hold sway. The push for mixed-methods research in AL in recent years has helped, but a divide still exists, and QCA represents an important approach to bridging that gap. As Ragin (2014) put it, QCA aims to build that bridge with an analytical approach based on “a case-oriented strategy of comparative research [in which] cases are examined as wholes—as combinations of characteristics … which distinguishes it from mainstream statistical methodology” (p. 16). Although QCA has been used for decades in a growing number of fields, including Sociology, Medicine, Civil Engineering, and Robotics research (see, for example, Chong et al., 2021; Eden et al., 2011; Song et al., 2022; Thomann & Maggetti, 2020), QCA still does not yet appear to have made its way to the fields of AL or TESOL. This paper, therefore, introduces QCA, briefly summarizes how it works, what distinguishes it from conventional analytical techniques, and how it might be employed in our fields.

Comparison of QCA with Conventional Analytical Techniques in AL

According to Pappas and Woodside (2021), who drew on Ragin’s initial work (1987), and who acknowledge the importance of context (Curtis, 2017), QCA is a “data analysis technique that combines the logic and empirical intensity of qualitative approaches that are rich in contextual information, with quantitative methods that deal with large numbers of cases and are more generalizable” (p. 1). Pappas and Woodside (2021) went on to explain that the ability of QCA to bring “together basic concepts from both qualitative and quantitative techniques of analysis differs substantially from traditional methods of quantitative analysis” and that QCA can “identify logically simplified statements that describe different combinations (or configurations) of conditions indicating a specific outcome” (p. 1). Therefore, QCA offers an alternative to the more traditional methods of analysis used in AL up until now, to explore the complexities of the foreign language classroom, beyond conventional frameworks and parameters. However, as with all methodologies, from teaching to research, and everything in-between, each one has its own strengths and weaknesses and its benefits and limitations, as one size does not fit all. QCA may, then, be most usefully viewed and utilized as a complement to traditional analytical techniques, such as regression analysis, correlation analysis, and structural equation modeling.

Specifically, regression analysis examines the relationship between a dependent variable and one or more independent variables, providing a quantitative estimate of their relationship and identifying significant predictors. However, it tends to assume a more linear relationship between variables and cannot establish causal relationships. Correlation analysis measures the strength and direction of relationships between continuous variables, providing a simple measure of association, but does not imply causation or identify non-linear relationships. Structural equation modeling tests complex relationships between observed and latent variables, allowing the examination of both direct and indirect effects within comprehensive models, but requiring large sample sizes for accurate estimation. Instead of those kinds of traditional analytical approaches, QCA identifies necessary and/or sufficient conditions and configurations leading to outcomes and provides a systematic approach to analyzing data, but relies on specific assumptions about causal complexity and configurational relationships.

Key Steps in QCA

There are a number of key steps in using QCA:

1. Formulate research questions or hypotheses that indicate the relationships between variables and patterns of causal complexity, i.e., situations and contexts in which there is not a straightforward one-to-one relationship between cause-and-effect. According to Ragin (2014), causal complexity is “a key characteristic of social life” (p. 28), which “is characterized by its conjunctural or combinatorial nature” (p. 25)”. An example of a common, recurring concern and question in education is what combination of factors, such as peer relationships, school climate, transportation accessibility, cultural values, and education policies, lead to high student attendance at the high school level (see, for example, Robertson-Wilson et al., 2008). The selected variables should be supported by a model, theory, or framework that connects the different variables in ways that make the relationships between them clear.
2. Select a configuration model or theory that will serve as a theoretical framework for the QCA analysis, representing the relationships between the variables identified in the research questions or hypotheses and providing guidance for the investigation. However, one of the challenges in QCA is how to categorize the different variables. As an example, Table 1 shows the Ecological Systems Theory developed by Urie Bronfenbrenner in 1979, which includes five dimensions—microsystem, mesosystem, exosystem, macrosystem, and chronosystem (Bronfenbrenner, 1979)—that can be used to categorize the different variables. Bronfenbrenner took an ecological approach to human development, which was considered ground-breaking at the time.
3. Although it is rare that any one variable does not connect with at least one other variable, making them all interdependent in one way or another, in QCA, there are condition variables (independent variables) and outcome variables (dependent variables). These variables form the basis of the QCA analysis and help define the scope of the analysis and determine which factors will be examined in relation to the outcome variable. As Table 1 shows, peer relationships (microsystem), school climate (mesosystem), transportation accessibility (exosystem), cultural values (macrosystem), and education policies (chronosystem) are condition variables, and high student attendance is the outcome variable.
4. Calibrate the data by assigning set membership scores—either zero or one—to variables to indicate their relevance or presence within a given case. This practice is based on Boolean Algebra, which utilizes binary zero/one variables. In QCA analysis, when a condition variable is zero that indicates its absence (or not fully included in the set), while a value of one indicates its presence (or fully included in the set). Calibration plays a critical step in QCA as it allows researchers to capture the nuances of each case by quantifying the degree to which a particular condition or outcome variable is present or relevant. By assigning set membership scores of zero or one, researchers can represent more complex phenomena, such as language teaching and learning contexts, and their relationship to specific outcomes.
5. In QCA, a “truth table” is an “analytic device …, which displays the data in a matrix of logically possible configurations of causal conditions” (Ragin, 2014, p. 23). The next step, then, is to create such a table to show all combinations of condition and outcome variables of interest, an example of which is shown below in Table 1. The truth table is the result of the calibration and will form the basis of the analysis that follows.
6. Use software to identify necessary and sufficient conditions and configurations that lead to the outcome variable across different cases. An example of such software is fuzzy set/Qualitative Comparative Analysis (fs/QCA) (Ragin & Davey, 2022). Here “fuzzy” refers to a value defined as being between zero and one, while also including both zero and one, and specifies the degree to which something is deemed to be “true.” The current version of the fs/QCA software is version 4.1. fs/QCA provides a user-friendly interface that allows researchers to input their data, define conditions and outcome variables, and perform different types of QCA analyses. The software includes interactive tools for data visualization, model specification, and interpretation of results, making it accessible to users with different levels of technical expertise.
7. Interpret the results in relation to the research questions or hypotheses, and consider the implications for theory, practice, and future research.
   
   

Table 1. Example of a Truth Table

Concluding Remarks

We have made a number of claims in this paper that are of potential significance to the fields of TESOL and AL research. First, we claim, with examples, that although QCA has been used for decades in a number of other disciplinary domains of knowledge, QCA does not appear to have been used in TESOL and AL research. The second claim is trickier to show, as it is not usually possible to show the absence of something that is not there. However, the recent launch of RMLA, focused on research methods in AL, confirms the need for more innovation in this particular area. Our third claim is that research in our fields could benefit from using QCA, perhaps in conjunction with more traditional methods, as an important addition to the methodological repertoire currently employed, which we briefly compare and contrast with QCA.

We also explain, in as much detail as possible in a shorter and more reader-friendly article, the key steps in carrying out QCA, using the generic example from general education of high-school attendance. Lastly, we realize that to claim to bring something “new” to any long-established field, including ours, runs the risk of being resisted and rejected, at least initially. But it is our sincere hope that by bringing QCA to TESOL and AL we are helping to advance research that can, ultimately, benefit teachers, students, and others in the long, complex, and challenging work of foreign/English language teaching and learning.

References

Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Harvard University Press. https://doi.org/10.2307/j.ctv26071r6

Chong, H.-Y., Liang, M., & Liao, P.-C. (2021). Normative visual patterns for hazard recognition: A crisp-set qualitative comparative analysis approach. KSCE Journal of Civil Engineering, 25, 1545–1554. https://doi.org/10.1007/s12205-021-1362-5

Curtis, A. (2017). Methods and methodologies for language teaching: The centrality of context. Bloomsbury Publishing.

Eden, J., Levit, L., Berg, A., & Morton, E. (Eds.). (2011). Finding what works in health care: Standards for systematic reviews. The National Academies Press.

Pappas, I. O., & Woodside, A. G. (2021). Fuzzy-set Qualitative Comparative Analysis (fsQCA): Guidelines for research practice in Information Systems and marketing. International Journal of Information Management, 58, Article 102310. https://doi.org/10.1016/j.ijinfomgt.2021.102310

Phakiti, A., De Costa, P., Plonsky, L., & Starfield, S. (2018). Applied linguistics research: Current issues, methods, and trends. In A. Phakiti, P. De Costa, L. Plonsky, &; S. Starfield (Eds.), The Palgrave handbook of applied linguistics research methodology (pp. 5–29). Springer.

Ragin, C. C. (1987). The comparative method: Moving beyond qualitative and quantitative strategies. University of California Press. http://www.jstor.org/stable/10.1525/j.ctt1pnx57

Ragin, C. C. (2014). The comparative method: Moving beyond qualitative and quantitative strategies. University of California Press.

Ragin, C. C., & Davey, S. (2022). Fuzzy-Set/Qualitative Comparative Analysis (Version 4.1) [Computer software]. University of California Irvine. https://sites.socsci.uci.edu/~cragin/fsQCA/software.shtml

Robertson-Wilson, J. E., Leatherdale, S. T., & Wong, S. L. (2008). Social–ecological correlates of active commuting to school among high school students. Journal of Adolescent Health, 42(5), 486–495. https://doi.org/10.1016/j.jadohealth.2007.10.006

Song, C. S., Kim, Y.-K., Jo, B. W., & Park, S. (2022). Trust in humanoid robots in footwear stores: A large-N crisp-set qualitative comparative analysis (csQCA) model. Journal of Business Research, 152, 251–264. https://doi.org/10.1016/j.jbusres.2022.07.012

Thomann, E., & Maggetti, M. (2020). Designing research with qualitative comparative analysis (QCA): Approaches, challenges, and tools. Sociological Methods &; Research, 49(2), 356–386. https://doi.org/10.1177/0049124117729700

Yu Tang brings together a decade of teaching experience in education with several years of industry experience from running a start-up company. Her research focuses on English Medium Instruction (EMI) in higher education, with a particular emphasis on factors influencing university students’ academic success in EMI settings.

From 2015 to 2016, Dr. Andy Curtis served as the 50th President of the TESOL International Association. Over the last 30 years, he has (co)authored and (co)edited 200 publications, presented to 50,000 language educators in 100 countries, and his work has been read by 100,000 language educators in 150 countries.