Part 5 of 7Problem hit

IFEval's official grader uses unseeded randomness

IFEval's reference grading code uses randomness in two places and never fixes it, so grading the same responses twice can give different results. Pinning both makes grading repeatable. Two prompts are still graded against the wrong character, and they stay in, disclosed.

Instructions using random detection
95 of 834
the plan had expected 31
Prompts graded on a substitute letter
2
they ask for # and !
My grading vs the original
541 / 541
prompts graded identically
Grading twice, both pinned
identical
byte for byte, separate processes

An error bar is only meaningful if grading the same response always gives the same grade. IFEval's official grader doesn't guarantee that, and fixing it turned up two prompts that are graded wrongly no matter what.

Using the official grader without changing it

The grader is Google's reference code for IFEval, copied into the project byte for byte at a fixed version. Its own driver script looks up each response by the prompt's text. That works with one response per prompt, but this study has several per prompt (repeat passes and sampled answers), so I call the grader's checks through a thin layer that grades one response at a time. It runs the same checks, including the same eight forgiving variants the loose grading uses.

To prove that layer grades exactly as the original does, I needed reference grades. The original repository ships GPT-4 responses to all 541 prompts but not their grades, so I made them by running the unmodified grader on those responses. A test requires my layer to reproduce all 541 prompts' grades exactly, and it does. Each run also regrades a fixed 20-response sample of that set inside its container before its numbers count, including a language-detection case. That catches a container whose grader, NLTK data or random seeding differs from the tested one. Every counted run cleared it.

Two sources of randomness

The grader never seeds either of the two random number generators its checks can reach. No file in it sets DetectorFactory.seed or random.seed.

The language detector, which was expected. Some checks call langdetect, a library that identifies a text's language by sampling pieces of it at random, so its answer can vary unless DetectorFactory.seed is set. The plan expected only the 31 instructions that ask for a response in a given language to use it. In fact three checks do, covering 95 of the 834 instructions: the response-language check (31), all capital letters (25) and all lowercase letters (39).

Python's built-in random, which wasn't. The check that counts how often a letter appears only accepts a single letter, a to z in either case. Given anything else, it picks a random letter with random.choice(string.ascii_letters) and counts that. Two prompts in the dataset ask for a character that isn't a letter:

  • prompt 1122 asks for a tweet with at least four hashtags, so #;
  • prompt 1129 asks for a pitch with at least six exclamation marks, so !.

For both, the grader counts a random letter instead. I found this while bringing the grader into the project: I logged every call to random.choice and random.randint while setting up each of the 834 instructions from its real settings, and exactly these two used random. With only the language detector seeded, grading the GPT-4 responses twice still gave different results for prompt 1122.

The fix, and what it leaves wrong

Before grading each response, strictly and then loosely, I reset both random generators to a fixed seed. A grade then depends only on the prompt, the response and the instruction's settings. Grading all 541 GPT-4 responses twice, in separate processes, gives byte-identical results. With only the language detector seeded, it didn't.

That makes prompts 1122 and 1129 repeatable, but they are still graded wrongly. With the fixed seed, the substitute is the first letter drawn after random.seed(0), which the checker counts as y. So the grader checks whether the tweet contains at least four ys and the pitch at least six.

I drop neither prompt, so the scores stay comparable with other IFEval results, and disclose the bug here instead. Both prompts ask for "at least" some number, and every loose variant only deletes text, so their strict and loose grades on this instruction always agree. The bug therefore can't affect the strict-versus-loose comparison. It can affect every other number that uses strict grades, including the main comparison, and only through these two prompts. Together they can move a model's score by at most 0.37 points and the gap between the models by at most 0.74, against an error bar of about ±4.

Other behavior of the original, kept as it is

  • Strict grading marks an empty response wrong.
  • Where the language detector can't classify a text and raises an error, the three checks that use it treat the language instruction as followed.
  • Prompt 2785. The Hugging Face copy of the prompt asks for "at least one placeholder", and the grader's own GitHub copy asks for "at least 3". Both carry a setting of three placeholders, so the Hugging Face wording contradicts its own setting. The grader reads only the setting, so a response is held to three placeholders when the prompt asked for one. I use the Hugging Face copy, which is the pinned source, and keep the prompt.