Are resume match scores accurate?

Mostly no — and not because the AI is weak, but because most tools re-decide what the job requires every single time they score you. When we ran the same résumé against the same job description three times through a naive scoring pipeline, it returned 60, then 55, then 50. Of the 15 requirements the tool extracted from the JD, only 8 appeared in all three runs. The score was not measuring the résumé. It was measuring the tool's own mood.

Why the same résumé gets three different scores

Every LLM-based matcher has a hidden first step: read the job description and decide what it requires. That extraction is itself a language-model call, and it comes out different every time. In our logs, one run called a requirement "Social", the next run called it "Social media marketing", and the total requirement count drifted between 14 and 15. If the checklist changes under you, the percentage on top of it is noise with a confident font.

We also measured the noise floor directly: rerunning the identical input produced scores spread across a ±10 point band. That band is wider than the improvement most people get from a full rewrite session — which means an unpinned tool can tell you your edits made things worse when nothing changed at all.

The fix is boring: pin the requirement list

Read the JD exactly once, in a call that is not allowed to see the résumé (otherwise the candidate influences what the job supposedly requires). Freeze that requirement list. Every later score — first diagnosis, after each edit, final check — reuses the same frozen list and the same grader. After we made that change, the same input scored 30 four times in a row. A lower number you can trust beats a higher number that changes when you refresh.

Other ways a match score can lie to you

• Total years counted as specialty years: "5–10 years of product management" is not satisfied by 9 years of anything — total experience is always ≥ specialty experience, so quantity-only parsing always flatters you.
• Signals from a different domain multiplied in: we found a "writing quality" factor silently multiplying a job-match score, pulling a 100%-requirements-met candidate down to 65 for using too few metrics in bullets. Unrelated signals belong beside the score, never inside it.
• Two sources for one number: a headline that shows a fallback estimate while the real engine is still computing will visibly change from 59% to 70% on screen. One public number must have exactly one source.
• Optimistic aggregation under failure: a "median of 3 samples" quietly becomes "the higher of 2" when one sample fails. Whenever the optimistic side wins ties, the score inflates over time.

What to do with any match score

Treat the percentage as a direction, not a verdict, and ask the tool two questions before trusting it: does the same input give the same number twice, and can you see the frozen requirement list the score is computed against? If either answer is no, use the itemized requirement breakdown and ignore the headline number. These two failures are why we built OfferCoach with a pinned rubric and a single scoring path — and if the engine can't compute a score, it says "can't compute" instead of guessing.