There are various opinions on this topic, and I researched it a few years ago, using old contest programs from division contests and higher. With ten contests in a few days, and 88 contestants, the 2013 semi-finals and finals looked like a great opportunity to revisit this.
The data shows a decisive bias, speakers later in the order are more likely to win. Out of 30 placings (1st/2nd/3rd in 9 semis plus the final), the average place in the speaking order was 60% of the way through. If speaking order had no impact, this would be 50% (about the same as I found in my previous analysis).
First place was especially biased, at 72%, over 2/3 through the contest. In the ten contests, two first place winners spoke last, 8 of the 10 spoke in the last half. Second place was at 52%, third place at 58%. Only 1 first-place winner (out of 10) was among the first 3 speakers in a contest, only 1 second-place winner was among the first 3, and only 4 third-place winners were among the first 3.
How do we address this? The only solution I see is in working harder to make sure judges are aware of this bias, in their training, and in their briefings. Some advocate independently scoring each contestant, without referring to the scores of previous contestants. It does tell us that we should not let contestants swap speaking order, even if both are willing.
Your ideas?
Full details in this spreadsheet: SpeakingOrder
Here’s the previous analysis which I had posted on LinkedIn in early 2012:
“This was mostly international speeches, there weren’t enough of the other types to get reliable numbers from them. I would doubt there’s a difference, but who knows. My sampling was division or higher contests, at least 4 contestants (but almost always at least 5).
The first place winner was, on average, 68% of the way through the order, second place was 65%, and third place was 54%. If there were no advantage, you would expect to see around 50% for all three (halfway through the order).
Out of 49 contests that I reviewed, 34 winners (69%) were in the last half of the order, while 14 (29%) were the last speaker. 23 winners (47%) were the last OR next to last speaker. Pretty clear edge.”
Just a thought experiment… I don’t know if this would be viable, but it may be a start to a solution…
Right now, the “scores” you give on the judge’s form are only for your reference; you still have to write down on the ballot your choice for 1st, 2nd, and 3rd.
Now, even if we just go with the “scores” there may still be a bias, since they may give higher scores to the later speakers…
What if we use MORE judges in the audience, each one assigned to 1 or 2 speakers. They give the speakers a SCORE. Since each judge is not giving a score for ALL contestants, there should be a much less chance for a bias on the later speakers….
Interesting idea, but I think most contest chief judges already have a major challenge in finding just the minimum number of qualified judges.
The other problem is that the scoring is inherently subjective — for example, how many points (out of 15) do you take off for “speech value” that doesn’t quite hit the right spot for you? It’s really not possible to compare those raw 0-100 scores between judges, and there’s no way to make them consistently comparable.
Only the ranking is useful.
Why start on the assumption of 15 and work back? Surely we should start at 0 and work forward? Anything else brings too much cognition into play and I feel we should judge spontaneously on experience and impression. The moment we start to think we confuse the issue. Does anyone have any psychology theory to impart?
That was just an example, and adding or subtracting points is still subjective, there’s no way to provide guidelines on just how many points you add/subtract for each category.
Judging “spontaneously” sounds like you’re encouraging subjectivity — and makes the fading memories of the speech an hour ago even more influential.
If you want to judge on “impression”, then perhaps we should throw away the guide entirely, and just let the judge award a single numeric score based on their impression?
Hi Mike,
When judging you have 60 secs to write down the marks.
I do not get why you correlate spontaneity with subjectivity or impression with throwing away the guide.
What I mean is we as humans have listened to the speech and formed an impression of it, indeed it is the task of the speaker to create that impression upon us.
I believe we are best equipped to judge the impression spontaneously, in other “in that moment.” Thus if we ask ourselves a score out of 15 we come up with a simple answer as opposed to some system whereby we start at 15 and then do this and this and…this ultimately brings too much cognition into a simple act.
Anyway, if I am totally honest I feel the simple reason latter speakers win is because there has not been a standout early on and we WANT someone to excite us later because to be honest there are very few great speeches even in the International finals. Many of my friends cringe listening to even World Championship speeches which are often one person dramatic acts than cleverly and effectively worded moments to change our lives.
Rob, fair point. I’m just saying that in the end, it’s a very subjective decision how many points to award in each category. There’s no practical way to standardize that across judges, and so they cannot be compared between judges.
The whole speaking order bias arises when a judge remembers a speech differently an hour later and revises those marks.
I raised this last night and I actually judged last night. The more I think about it the more curious this gets. I did find it impossible to remember the earlier speeches in any meaningful way as the contest went on. One thing I did discover is maybe the sections need to be ungrouped to where 15pt sections focusing on three things become 3x5pt sections. With this we can judge on excellent, above average, average, below average, poor. May be a bit easier to be consistent, off topic, not sure, but I do hope you come up with some reason or cure. One last thought is I think speeches are light camera flashes, bright at the time but dimming with time, thus a recent flash is brighter than an earlier one.
I like the camera flash analogy!
Breaking out the scoring into more categories would slow down judging, requiring more time to assign a number in each category, as well as to add them up. Not saying it’s BAD, just that there is a trade-off.
Yep, true, maybe it would mean tally counters add up and so judges dont change results despite initial marks?! You only really need a tally counter without a tie anyway.
Dear Mike,
While statistics may prove that there is a relation between the speaking order, I personally feel that it is all about the delivery. There is a possibility that the speakers who are lower down the order gain confidence and learn from the past speakers and hence their deliveries improve.
I am not sure if we asked all participants to stay outside the room during others delivery, would there be any impact.
It’s possible, but I do not feel it is likely, especially at later levels in the contest progression.
This is definitely possible perhaps due to being better mentally prepared when not first or second. How do your stats work when you remover the first two participants?
I’m not sure what you mean, remove the first two participants? The bias is towards winning from the end of the order.
Let’s say there are 9 speakers. I would say by the time the third speaker starts the audience is more receptive and the speaker is better mentally prepared. So, if we consider that this could be so and we removed the number one and two speakers from your results would we see the same bias spread over speakers 3 to 9? We have assumed it is a judging bias but it could well be that the audience takes a speech or two to give that subtle feedback to the speaker improving their performance or indeed speakers just don’t get the juices flowing when competing in those early slots.
The middle speakers are still measurably less likely to win than the last speakers, so no, this wouldn’t help. It’s a generally continuous curve, the later you speak, the more likely you are to win.
A good few years ago now, the Director of TI, Terry McCann, attended the District 71 International Speech Contest finals in Ireland.
Before the contest started he looked over the list of contestants with me and pointed to the name of a person about two thirds down the speaking order and said “that’s the winner”. Obviously I said to him, “how come”? He answered, “because speaking position is everything at this level”.
As it happens, he was wrong, that person came second and the one that followed, won.
I don’t think I have entered a speech contest since that time! Not that I ever got to District Level (other than with topics) but it does show that however good the judges might be, human psychology wins out in the end.
If one were a betting man, betting on the last two speakers would often pay off.
When I first competed at the International finals in 1994, I was speaker #1 of 9, and I wasnt good enough to win. During the 1990s, two champions emerged from slot #1: David Ross (1991) & Dana LaMon (1992) More than 20 years ago, I know, and I encourage speakers who find themselves in slot #1 to be so good, that they set a high bar that seems impossible to overcome.
Speaking first certainly does not make it impossible to win, but it is tougher. Anecdotal stories will always bring out the exceptions, but the statistics are clear. On average, speaking later in the order improves your chances of winning.
I think many judges are completely wrong in how they judge. So often I have heard how we should mark the first speech lower and judge the others in comparison – this is nonsense.
You must judge the first speaker as if there were no other speakers. In fact to my mind you should judge every speaker as if there were no other speakers. You should also not be afraid to give perfect marks if indeed you thought they were “perfect”.
By the time you get to the fourth speaker, can you really remember what the first speaker was like, I think not. Therefore you formed a general impression, a fuzzy image of what they were like and went from there which disadvantages those early speakers.
No, if you judge everyone as if they were the only one then you have a better image because you are judging them according to your ideal which is less wavering than your memory of a earlier speaker.
When you get to the end you should add your marks up and see who won. Now it may well be that first, second or third is particularly close or even. Then and only then should you decide whether to change order (maybe there is no more than a point difference) but err on the side of caution if you do.
I have heard judges say that “the allotted marks (the judging sheet sections) are just a guide” – they are not! That is what every contestants considers when they compete and so should you. TM have thought long and hard about those criteria and you should not second guess them.
I used to be of the opinion that you HAVE to compare each speaker to the previous ones, and was skeptical of people who did origami on their ballot to hide previous scores.
With this order bias though, I think the only solution is to judge each speaker entirely on their own, and to never revise them at the end (though you may have to break your own tie scores sometimes, so avoid the perfect score too much!).
Whenever I am Chief Judge, before I start the “official” judges briefing (ie, going through the checklist) I generally suggest that they do the fold-over.
I encourage judges to mark each speaker individually, without any regard to the other speakers or their speaking order which is chosen at random. We go over the “Barriers to Objectivity” as we call them.
(Speaking order, judge as if first time hearing speech, don’t judge based on the fact that you heard them do it better before, it’s about how they perform tonight, etc.)
Sort of a mini “judges training”
But in the end, we’re all human. It’s subjective, but I wouldn’t have it any other way…. I wouldn’t want speeches to be judged by comptuers or robots…
I don’t think you can eliminate biases. You can minimize them, but even the most prudent judge who is determined not to allow for any bias, may still do so on a subconscious level….
I think it could be that if your scores are really close or tied, you are more likely to remember the later speakers more clearly than the previous ones. It could also be that speakers later in the contest just sharpen their game based on what they hear, or that the audience gets more responsive as they warm up, and thereby give the later speakers an advantage.
Correlation is not causation. And I still think that if you want to win you Gould listen to David Brooks. Be so good that the only question on the judges’ minds is who will come in second.
Henrik, possibly, but I think judging bias on order is the most likely cause, more than later speakers ARE better, or that the audience is more responsive (9-10 speakers is more likely to tire them out, I think).
But there’s one way to test this: Video ten contestants, show them to one group of judges in the right order (order of speaking), get the scores. Then show them to another group of judges in reverse order (but don’t tell them), get the scores. See if there’s a difference.
This has been discussed at length in our district, with no real solution except to keep reminding the judges that this bias exists. (There is also a bias for funny even if a speech isn’t very well organized, and there is a bias for well-known toastmasters in the district.)
I wanted to pipe in with something I have noticed is that speaking order does matter – but only when the competition is close. It has no relevance when there is a clear stand-out.
If there are 5 contestants, and one clearly is above and over the others, that person will win no matter when they spoke. But if two or three contestants are all very good, with no clear winner, those who spoke last have about a 90% chance of winning.
I believe that is what happened this year at international. I think if Kingi had gone last, and the one who won had gone 5th, Kingi would have won. It was that close.
There are two lessons to learn from this, I think. One, if you are a speaker, be above and beyond everyone else to win. And if you aren’t above and beyond everyone else, speak last 🙂
Yes, you CAN win from any position. But if you’re only SLIGHTLY better than the rest, you won’t do so from the front half, you have to be noticeably better.
On the other hand, you could be SLIGHTLY worse than the rest, or at least no better, and still win if you’re at the end.
Some more data, looking JUST at the World Championship, the final round, over the last nine years, there’s a clear bias towards the last speakers winning:
2005, winner was 8th slot (of 10)
2006, 6th slot (of 10)
2007, 8th slot (of 10)
2008, 9th slot (of 10)
2009, 8th slot (of 10)
2010, 5th slot (of 9)
2011, (don’t know)
2012, 8th slot (of 9)
2013, 9th slot (of 9)
Some clever statistician would smile looking at those outcomes wouldn’t they 🙂
There’s another possibility I thought about…
What if there’s another factor besides a judge’s bias regarding speaking order? Perhaps sloppy habits as a judge give the later speakers a chance to figure out who the judges are, giving them an advantage.
Something like this could even be happening unconsciously — If you notice a judge, you will be more attuned to their connection, and your speech will reflect it.
During that minute of silence, the judges mark their ballots, but they have to be careful so that the contestants don’t know that they are judges. I see a lot of judges who mark their ballots openly, so you can’t miss them…
I’m not saying this is the reason for a bias toward later speakers, but may be a contributing factor…
Even if a contestant knew who the judges were and where they were sitting, what could they do to improve their score? They might know one judge thinks doing X is important, but I don’t see how that would extend to many judges, and how a contestant could realistically modify their speech on the fly to include X (for every judge), and still end on time with a coherent organized presentation.
Not a factor at International level.
I don’t mean they actually try to do anything differently, but it does have a psychological effect, and can be subtle… On both sides… For the speaker, it can be that unconsciously they pay more “attention” (eye contact?) with judges, and on the Judge’s side, they may unconsciously “pick up” on that extra attention, giving them a feeling of familiarity when they’re making their decisions…
Just a possible “contributing” factor ..
Certainly possible, but I think much smaller than the “good old days” effect of remembering a speech from an hour earlier, after 6-8 other speakers have been up.
Easy way to test this though — we need ten speech videos and two pools of highly qualified judges.
The suggestion that judges would look more favorably on a contestant on the basis of a little eye contact doesn’t say much about the integrity of the judges.
The more contestants (usually the higher the level, but not always), the more bias is likely based on order. Having competed at the WCPS twice, once in 7th position, when I took 3rd, once in 1st position, when I did not place – I’d have to say it did have an effect.
That said, my speech the year I did not place didn’t deserve to place, either. It might have been a better speech had I had an extra 50 minutes watching the other speakers – I’ve changed things in speeches in the past when given time, and, personally, I gain energy as I watch the others.
Of course there’s also the audience, which is much more warmed up later in the contest, whose reaction to a speech can affect the judging.
It’s interesting to me that a winning speech becomes better than it was after it wins. Suddenly most people just assign it greatness, and elevate higher than all the others based on the decision of a judging panel vs. their own feelings as they watched the contest. But that might be fodder for a different post.
Bottom line – yes, I think order affects the results – especially if the contestants themselves think it does.
Thanks, Rich, good observations! The worst part may well be the contestants psyching themselves out when they draw what they think is a bad slot.
My experience is there is an impact, but it is based on field and speaker personality.
The larger the field, the more speaking helps. I think with a 5-6 person field the matter of primacy and recency even out. No one has gotten bored, or is too tired or unfocused, to stay tuned in the entire time. Once you get to 8 or beyond, especially if the contest is early in the AM, the more chance you have for audience (and judges) to stay focused.
By the same token, the mere idea of speaking first causes some speakers to dry up and blow away. I was in a district international final with a TM who had been on The Big Stage. We drew our numbers, he was first. His reaction, “Well, now I’m the warm up act”. He did not win…I think in no small part because he had convinced himself he was not going to win.
Yes, this only becomes an issue at big contests, like 8-10 speakers. Unfortunately, the most critical ones are just those — district (some districts), semi-finals, and finals.
Yet going in 2nd or 3rd position out of a big field is only slightly more likely to win than speaking first. If there’s a warm-up effect, it stretches over many speakers, not just the first one, and then is lost in the recency effect, which is quite similar.
The conclusions I draw are thus, please feel free to add:
That judges should judge each contestant as if they were the only one and judge them against a perceived “ideal” speech.
That speakers may psyche themselves out by drawing early.
That audiences warm up as contests go on and may even become more engaged and applaud more as the contest nears it’s finish subjectively affecting the judges perception of the speech and how it’s being received.
That earlier speakers may be forgotten or marked less generously to allow ‘room’ for ‘better’ speakers later.
Thus the recommendation would be to limit contests at major events to 6 and/or have a warm up speech or two prior. Neither of which are without issue.
What do others feel? Mike do you have any ideas of what we should do given this bias? Can we change the way we train judges or organise contests?
Rob, that’s a great summary. And my proposal is to train each judge to mark against an ideal speaker on the published criteria, not the previous speaker, and help reinforce this by picking up scores after each contestant.
I don’t think there’s a significant warm-up effect, audience reaction appears about the same for similar-quality speakers whether first or last, but that’s my subjective opinion.
Thanks for a great discussion topic and all the various opinions, excellent to thrash these ideas out, that’s my subjective view 🙂 Keep it up!
Thanks, Rob!