myTESOL Lounge

Expand all | Collapse all

Test Prep and TOEFL Validity

  • 1.  Test Prep and TOEFL Validity

    Posted 16-01-2018 09:22 PM
    Edited by Steven O'Loughlin 16-01-2018 09:25 PM
    Hello all,

    I'm currently doing research on the validity of the TOEFL test in relation to developing communicative competency for academic studies at the tertiary level. My focus is specifically on the influence of test preparation courses in Asian countries that emphasize high-stakes testing as a gate keeper to greater educational opportunities.  I am looking for any feedback or knowledge of research from fellow English language teachers or internationally-focused educators who have experience or knowledge of this topic.   Several themes I'm seeking to understand are as follows:

    1. Are there test preparation strategies and techniques which potentially compromise TOEFL validity as a valid indicator of English proficiency?
    2. What is the impact of test preparation courses on students communicative competence in the academic domain?
    3. What actions are ETS (the test creator), colleges, and universities taking to ensure that the TOEFL test remains a valid indicator of students ability to succeed at the tertiary level?

    I currently work as the director at an Intensive English Program in China with an EAP curriculum.  Over the last ten years, I have worked with many students who enroll in our program after taking test-prep courses in China and whom remain unable to communicate effectively in English in speaking or writing.  This is despite having obtained an TOEFL score (70+) that would meet the threshold for many universities and colleges in the USA.  My goal is to develop a comprehensive understanding of the topic and how it is being addressed in the language teaching and international education community.

    Any comments or direction is greatly appreciated.

    Sincerely,

    ------------------------------
    Steven O'Loughlin
    Academic Director

    ------------------------------


  • 2.  RE: Test Prep and TOEFL Validity

    Posted 16-01-2018 10:41 PM
    Hi Steven,

    Please allow me to be among the first to commend your current endeavour to explore high-stakes test prep. in relation to communicative competence in tertiary-level academic settings.

    Predictive validity of tests such as the TOEFL or TOEIC, particularly with regard to students simply cramming for them intensively yet shortly in advance, is a question that long has been on my mind.

    It sounds to me as if you would define academic success at least in terms of abilities to speak and write English for academic purposes (EAP), which leads me to wonder how the EAP curriculum under your direction in China defines such purposes and assesses their achievement.

    I sincerely hope you will synthesize your definitions and findings, and share them with us on a regular basis.

    Best regards,

    Paul

    ------------------------------
    Paul Beaufait
    Associate Professor
    Japan
    ------------------------------



  • 3.  RE: Test Prep and TOEFL Validity

    Posted 17-01-2018 04:24 AM
    ​Hi Steven
    I am glad to know that you are exploring test preparation and its predictive validity. Are you also looking at the relationship between strategy type and their test scores?

    Actually, I am also looking at the test preparation strategies for a national level test in Nepal. In my study, I am focusing on Particular strategies that are directly related to language learning strategies. I am following Xie (2013, 2015) and also following Messick (1982).

    I am attending and presenting at the TESOL convention this time. I would love to see you there, if you are also coming.
    Warm regards,
    Saraswati

    ------------------------------
    Saraswati Dawadi PhD student
    Mrs
    The Open University, England
    United Kingdom
    ------------------------------



  • 4.  RE: Test Prep and TOEFL Validity

    Posted 17-01-2018 11:55 AM
    These are important questions, Steven.  I'd be eager to see your results.

    ------------------------------
    James Radebaugh
    Clark University/Hefei University of Technology
    ------------------------------



  • 5.  RE: Test Prep and TOEFL Validity

    Posted 18-01-2018 04:21 PM
    Edited by Lauren Rein-Hamberg 19-01-2018 09:44 AM
    Hello Mr. O'Loughlin,

    Our university has a 2+2 partnership with a university in Shanghai. We've found that TOEFL (and IELTS) are not strong predictors of students' academic abilities. The students who come with the minimum score to enroll in academic classes sometimes lack the academic skills to be fully successful. I did a study comparing GPAs of those students who did and did not go through our Intensive English Program, showing that even one session of English training noticeably improved GPAs.

    In order to address this, we've started both an online and in-class program to orient students to U.S. academic culture, speaking, and writing. The online course can be offered any time to college or high school students. More information here: CIEP English & Culture Immersion Program

    I can share with you both the GPA research and the information on the online course in case you are interested in seeing how our IEP and university is addressing this issue. I can be reached at lauren(dot)rein(at)uni.edu. Hopefully your colleagues and students will find the study and online course interesting as well.

    I look forward to hearing from you.

    Yours,
    Lauren Rein
    (Curriculum & Assessment Coordinator, CIEP, University of Northern Iowa)





  • 6.  RE: Test Prep and TOEFL Validity

    Posted 18-01-2018 11:52 PM
    Edited by Steven O'Loughlin 18-01-2018 11:52 PM
    Hi Everyone,

    Thanks very much for all your contributions and I have a lot to consider based on your input.  One resource I can point out in regards to the TOEFL test is the ETS website which provides access to a substantial amount of research regarding the development of the test. (http://www.ets.org/toefl/research/). One paper that has been influential to me over the last several years is a quantitative survey of students' and faculties' perceptions of tasks important for academic success, particularly in relation to the importance placed by both groups on academic listening.

    Rosenfeld, Michael, Susan Leung, and Philip K. Oltman. The reading, writing, speaking, and listening tasks important for academic success at the undergraduate and graduate levels. Educational Testing Service, 2001.

    ------------------------------
    Steven O'Loughlin
    Academic Director

    ------------------------------



  • 7.  RE: Test Prep and TOEFL Validity

    Posted 18-01-2018 10:26 PM

     

    Hi Saraswati,

     

    Thanks so much for you input and references.  Can you give me more details about the Xie reference? I'm having trouble locating the source through the university database.  I also appreciate the Messick reference as I wasn't aware of this framework. I have been using Bachman's framework almost exclusively.  I guess I still have a ways to go before understanding the topic comprehensively.

     

    Unfortunately, I won't be going to Spain for the conference. However, if you are aware of any upcoming conferences in Asia, or know a good way to stay informed about them, I would be very appreciative. 

     

    Best Regards,

     

    Steve

     

    Sent from Mail for Windows 10

     






  • 8.  RE: Test Prep and TOEFL Validity

    Posted 17-01-2018 10:50 AM
    We have been carrying out some internal research on the question of the value of standardized tests such as TOEFL and IELTS compared to at least one term in our intensive on-campus university-entrance focused EAP program, and have solid data demonstrating that the EAP program is a significantly better predictor of university success than the tests. It's also a slightly better predictor than completion of English-medium high school programs. Of course, the comparisons are a little "<g class="gr_ gr_331 gr-alert gr_spell gr_disable_anim_appear ContextualSpelling ins-del" id="331" data-gr-id="331">apple-and-oranges</g>" because the on-campus EAP program isn't only teaching academic English; it's also providing orientation to the campus, the community, and Canadian university expectations. However, when you consider how much some people end up spending on TOEFL preparation courses, repeated testings, and tuition for failed courses once they are studying in their selected English-medium postsecondary institutions, the comparisons do provide evidence that expenditures related to the standardized tests may not be as good an investment of time or money as a term or two in a well-designed EAP program.

    ------------------------------
    Shannon M. Storey
    Academic Coordinator, Certificate in TESL
    University of Saskatchewan
    Saskatoon, Canada
    ------------------------------