SpeechRater™ charts: Interpreting the graph

The ETS® SpeechRater™ scoring engine collects a lot of speech-related data from TOEFL® Speaking responses.

Every response that is recorded on My Speaking Score and submitted for analysis is sent to SpeechRater - ETS's speech assessment software - for scoring.

On My Speaking Score, when you record a response and click get SpeechRater™ score, you access a tremendous amount of information about your TOEFL Speaking performance.

Background

Millions of people have taken the TOEFL from over 200 countries. And SpeechRater, with the help of human raters, assigns a score to every TOEFL Speaking response. Each response gets combined into a score on 30 using a secret calculation method.

If you're in the Good range - you have an overall TOEFL Speaking score between 26-30. The fair range, between 18-25. Followed by the Limited range, between 10-17, and the Weak range, between 0-9.

To calculate a TOEFL Speaking score, SpeechRater extracts all kinds of data from your responses.

My Speaking Score uses that SpeechRater data to show you how your response compares to data from all those other responses.

Every response you deliver on My Speaking score generates its own unique report.

SpeechRater™ Reports

Every SpeechRater report has 12 charts. Below is the Speaking Rate graph. Speaking Rate is one of 12 dimensions - and it's one of the most important dimensions.

You are here

So, back to our original challenge: how to you read this chart?

Notice the black vertical You are here line.

That indicates how this response compares to all other responses for this particular dimension.

Remember how SpeechRater collects all that scoring range data? The scoring range is found on the y axis.

The coloured horizontal bars line up with the Speaking Rate percentiles on the x axis. (My Speaking Score calls this your "score" - it's really a percentile.)

So now we can compare Speaking Rate performance with overall TOEFL Speaking score performance.

Because we have all that data, we know that responses with an overall TOEFL Speaking score in the Fair range, have Speaking Rates that are between the 33rd and 74th percentile.

That's a fact. Mouse over any horizontal bar to see the percentile range.

Percentiles

For the response below, the Speaking Rate is in the 54th percentile. #learnmore

This give us two talking points.

First, to earn a TOEFL Speaking score score in the Fair range - i.e. 18-25, you probably won't have a Speaking Rate below the 33rd percentile. Going below the 33rd percentile in Speaking Rate likely puts you in a lower score range.

Second, if your Speaking Rate is above the 74th percentile, you're very likely to have an overall TOEFL Speaking score in the Good range, indicated by the green bar.

Overlapping bars

There's also a percentile range where some speakers scored in the Good range, and some speakers scored in the Fair range.

You'll see this overlap all the time.

Suppose a response had a Speaking Rate in the 73rd percentile. 

You wouldn't know, based on this dimension alone, whether the response would be Good or Fair. 

In fact, the greater the overlap, the less reliable the dimension is as a predictor of overall TOEFL Speaking score.

It's safe to assume that dimensions with less reliability have a smaller impact on the overall TOEFL Speaking score. 

Vocabulary Diversity is less important than Speaking Rate, for example.

Recall that overall TOEFL Speaking scores are calculated using all kinds of data points.

And don't forget the human rater.

So you can't use one dimension from one response to predict someone's TOEFL Speaking score. It's also a mistake, for example, to think of overall TOEFL Speaking scores being neatly distributed along a horizontal bar. 

That's not how the ranges correlate.

For example, it's possible for a response in the 73rd Speaking Rate percentile to belong to a speaker with an overall TOEFL Speaking score of 18 - the lowest band of the Fair range.

Correlation, not causation

TOEFL Speaking score ranges and Speaking Rate percentile ranges correlate. There's a pretty strong correlation between the Speaking Rate dimension and overall TOEFL Speaking scores. 

The Speaking Rate of a response, therefore, is a pretty good predictor of an overall TOEFL Speaking score range.

In fact, if you only knew your student's Speaking Rate, you could make a decent guess about their overall TOEFL Speaking score range, provided you had enough speech samples.

Readability

My Speaking Score presents SpeechRater data in this way because research* has shown that feedback presented this way helps teachers and students identify aspects of TOEFL Speaking that need improvement. 

* Lin Gu, Larry Davis, Jacob Tao & Klaus Zechner (2021) Using spoken language technology for generating feedback to prepare for the TOEFL iBT® test: a user perception study, Assessment in Education: Principles, Policy & Practice, 28:1, 58-76, DOI: 10.1080/0969594X.2020.1735995