Is the Duolingo English Test a game changer?

Millions of people sign up for tests and exams each year to certify their level of English. The demographic of those looking to gain scores from B2 to C2 is clear. The majority are from 16-30 years old and are in, or wish to be in, higher education. Success in these exams opens doors and offers the chance to improve prospects. It provides young people with increased mobility and more options in employability and education.

Attendance at a test centre to carry out a test, whether paper-based or computer-based, has always been the standard way of proctoring the process. However, if candidates are unable to attend test centres to carry out exams because of restrictions, such as happened during the Covid pandemic, then the field becomes vulnerable to disruption.

Exam providers are mindful of this and have been using online proctoring systems for some time now. But there is a relatively new player with skin in the game. Enter Duolingo English Test (DET), stage left. The Duolingo app has been a popular addition to the language learning mix for many years. What has been more surprising in recent years is the success of their computer adaptive test for assessing English proficiency. Currently accepted by over 3,500 institutions, it provides a very user-friendly, convenient option which can be carried out at home instead of in a test centre. And let’s make no bones about it; it’s also really cheap.

I’ve been swotting up on this test, and its potential as a game changer, for a while now and I have some observations I’d like to share. I recorded a short video for the PASS YouTube and TikTok channels which I’ve embedded here too as some of the points are aligned with this blog.


As an adaptive test, DET is more comparable in genre to Linguaskill or Aptis than a standardized test such as IELTS or TOEFL iBT. Given how adaptive tests are now widely recognized by educational institutions, it begs a series of questions about what language testing is for and how it will develop in the future.

On PASS exam preparation courses we place emphasis on lifelong learning. We want students to take away transferable skills, such as in essay writing or study skills. We spend a lot of time on this on our courses because the frameworks learned as part of the exam preparation process will impact positively on later academic studies. So does computerized adaptive testing (CAT) cover enough ground to allow for this?

In the DET test readiness guide, we read that extensive study and preparation is not necessary, “No need for memorization, rehearsal or expensive classes”. It is also claimed that, “Intense studying right before the test will not change your score”. However, this year DET started scoring a previously unscored “Writing Sample” question, using a grading engine, and they added 2 sets of “Interactive Reading” questions. All in all, some pretty big changes. In order to accommodate these they have reduced the frequency of some other question types.

I studied the technical manual from the research part of the DET site, in which the scoring system is explained. It’s enlightening in itself but still leaves an impression of a rather opaque system. Other providers publish detailed descriptors which teachers can learn, understand and use to prepare candidates. They provide a lot of support in this area.

With adaptive tests it can be hard to know what the grading decisions are predicated on. The newly-scored DET Writing Sample is a case in point. This is a neat way of introducing a mini essay topic. The test taker is instructed to treat the task as a “short essay”. Immediately, alarm bells ring for an old-fashioned guy like me!

What are the grading criteria exactly? Where are the detailed descriptors? How can you learn how to do this task well if you don’t know exactly what they want from you? How can you do this task well if you don’t study and prepare it beforehand? The first doubt I had was regarding the level of formality and the register. Essays, however short, need to demonstrate awareness of register. So I contacted DET via the chat on the website. Basically, the upshot is that test takers are not penalized for using informal language, such as contractions, personal pronouns and direct questions.

However, this writing task will be seen by the institutions which the test taker sends their results to. Two of the tasks are included in the submission, this writing task and an unscored speaking task in video format, along with the scores on the certificate. Would the academic institution view this sample negatively if the writing was informal in register? Even though the grading engine scored it highly enough to pass the threshold for that institution.

I’m currently developing a PASS course for the DET and have decided to err on the side of caution. I will recommend using neutral or semi-formal language, bearing in mind that this writing sample will be viewed by people who make decisions on admissions. In addition, using some more formal language will set the tone for test takers to use more complex structures in their writing. After all, a grading engine if it’s any good at all will be programmed to know that passive voice is a little more complex than using active voice, or at least to recognise the variety of structures. It seems to be a win-win situation to take this line.

While carrying out practice exercises, I was further struck by something about the new Interactive Reading question types. Not much use is made of distractors in multiple choice options. Test takers will be delighted! Distractors, from a coherent lexico-grammatical set, are used widely by other exam providers in reading tasks to allow candidates to demonstrate their competence. They are considered to be an essential part of the metacognitive process. To put it in simpler terms, otherwise it might all be a bit too easy!

To take a final example, the “Read and Complete” task requires test takers to complete unfinished words in a text. I read about this activity in the technical manual but that didn’t prepare me for seeing the same unfinished words repeated in the task when I did the practice tests. Perhaps the task has a psychometric function, however, there were also words which needed to be completed which appeared in their complete form elsewhere in the same text. It’s hard to see the value of this activity for university graduates and undergraduates, though it may have application in other ambits, and maybe at lower levels. This is, after all, a multilevel test and presumably tasks are weighted.

I’m really looking forward to seeing how this test changes in the future, especially in terms of grading transparency and task development. The DET provides a window into a new way of assessing levels of English proficiency. The user-friendliness and UX of the website is exceptional, and the way online proctoring is carried out is innovative.

In order to prepare for the test, Duolingo stresses the need to immerse oneself in English. If this preparation were supplemented by more focused and targeted work on strategies and skills for each question type it would surely be beneficial. The Pass Duolingo English Test course will be launched on 13th June to cater for this need.

Sources and research:

https://www.cambridgeenglish.org/Images/526186-research-notes-72.pdf

https://englishtest.duolingo.com/research

https://englishtest.duolingo.com/resources

https://englishtest.duolingo.com/guide

https://englishtest.duolingo.com/quick


Location: Spain

Comments

  1. James Dunne16 June 2022 at 16:36

    Hi! Just a reminder that there's a Duolingo English Test playlist on the PASS YouTube channel: https://www.youtube.com/watch?v=V4Ac4EaovNo&list=PLA-iGWqIJKqOwLDDUYyxUXtjvY4kmlCg_
    🙂

    ReplyDelete

Post a Comment

Popular posts from this blog

IELTS and Cambridge exams - Computer-based or Paper-based?

Image
        Is the computer-based format better for IELTS and Cambridge exams?   All teachers who have worked in the field of English exam preparation know this scenario. Your student comes back from their exam looking a bit glum and you ask them what went wrong. They complain that they couldn’t concentrate because people were coughing and sneezing or there was building work going on outside. Moreover, typically, they say the sound system at the venue was poor quality and they couldn’t hear the audio recordings properly. Furthermore, the tables weren’t close enough together so it was almost impossible to cheat. Well, ok, I made up that last one! When students complain about exam conditions I usually reassure them that it probably won’t have had a big effect on their overall performance, and maybe I privately think they are getting their excuses in early. However, at the back of my mind I do feel a lot of sympathy as regards the quality of the audio. It really c...
Read more

How to get from B2 to C1 – What’s the difference in vocabulary?

Image
What vocabulary does a learner need to know at C1 level? How many words do they need to know? What’s a lemma? Does it matter? A lot of students worry about which vocabulary, and the amount of vocabulary, they should know at each level of English, especially when they’re getting to higher levels such as B2 and C1. I’d like to highlight a few key tips for learning vocabulary when moving from B2 to C1 level. Here’s the million dollar question in terms of exam preparation. Is it more important to learn all the words that correspond to C1 level, or to be able to use fewer words and expressions more accurately? Let’s rewind a little. I did a blog just before Easter asking the Easter bunny for three gifts to help with exam preparation. One of them was about how to structure a vocabulary notebook. I wrote that it should be based on collocations, with example sentences, and be organised by theme, part of exam or some other personal system . You’ll notice there was no mention of a numbe...
Read more

The Elon Musk chip that will revolutionise language learning! Or will it?

Image
Back in the nineties, I was living in Florence. A friend of a friend came out for a visit and was having a drink with a bunch of us language teachers. To the consternation of all present, he openly admitted he had no interest in learning languages and that he’d be delighted if someone could come up with a device that would do the work for him. In those days this kind of device was conceptualised as a machine that would translate your speech into the other language via an electronic voice. Lovely! We debated the pros and cons of this but couldn’t really come up with a strong argument against it, at least not strong enough for him, in spite of the fact that it could potentially put us all out of work. Well, über-magnate Elon Musk announced last year that a chip was in development that could revolutionise the way we communicate. The chip would be implanted into the brain and we would all communicate in a   universal language. It was only 5 years away! Great, right? Maybe it’ll com...
Read more