Margin-triggered answer reranking
I fine-tuned RoBERTa to 84.28 EM / 90.93 F1 on SQuAD, then raised it to 84.40 / 91.04 with a second model that only steps in when the first is unsure. It changed 55 of 10,570 answers, fixed 14 and broke 1.
01The problem
A second model that re-checks every answer (a reranker) fixes some and breaks others. I wanted to see if it could step in only where the first model was unsure, and leave everything else alone.
02What I did
All of it, on my own: the fine-tuning, the evaluation and the rule for when the second model steps in.
03What made it hard
SQuAD is a fixed public benchmark, so any answer I broke would show up straight away in the score.
04How it works
The first model's top pick is right for 8,908 of 10,570 questions, and its second pick is right for 661 more. When the second pick is the right one, the gap in confidence between the top two is much smaller (a median of 0.113 against 0.602). So the reranker only steps in when that gap is small, which happened 55 times, and every other answer passes through untouched.
05Other options I ruled out
Reranking every answer, which I tried first and which broke too many. I also searched the threshold settings systematically instead of picking them by hand.
06How I tested it
The standard SQuAD scores (EM and F1) against the baseline. Because it only touched 55 answers, I could check every one: 14 fixed, 1 broken, 40 still wrong.
07Results
EM went from 84.28 to 84.40 and F1 from 90.93 to 91.04, all from 14 fixes against 1 break. You can reproduce it from the repo.
08What's next
For 4.9% of questions the right answer isn't in the first model's top 5 at all, so this approach tops out at 90.5 EM. The idea I took away, only stepping in when unsure, also shows up in Select to AI's confirmation steps and GUS's final check.