Why does the same device score differently on two runs?

Run the test twice on one device and the numbers move. Some of that is measured and shown; some is not solved. What was found running the same machine four times in a row, what the pass spread does and does not tell you, and how to compare runs honestly.

Run the test twice on the same device and you will not get the same numbers. Some of that difference is measured and shown to you. Some of it is not understood, and this page says so, because a benchmark that pretends its numbers are exact is lying about the one thing it should be honest about.

What was actually found

One machine, four full runs back to back, nothing else changed. The power rating moved by a single point — 30, then 29, 29, 29. The CPU score moved by 2.4%. The average frame rate moved by 8.9% between runs — while the three passes inside each run agreed with each other to within one and four percent.

That last pair is the interesting one. Three passes a few seconds apart agree closely. Whole runs a minute apart do not agree as well. Something changes between one invocation and the next that three back-to-back passes cannot average away, and it has not been identified. Candidates include which cores the browser hands the work to, what else the device is doing, the state of the just-in-time compiler, and heat left over from the previous run. The honest summary is that the within-run figure is understood and the across-run one is not, yet.

What the pass spread tells you, and what it does not

Under the CPU score and the render test there is a caption reading "passes differed N%". Each subtest runs three times and that is how far the three results were apart. It is a genuine repeatability figure: it says how much you should trust the number above it.

It is a within-run figure. It does not tell you how far the next run will land from this one, and it must not be read that way. A spread of one percent inside a run does not mean the next run will be within one percent of this one; the measurement above says it may not be.

Where the movement comes from

• What else the device is doing. An app syncing in the background, a notification arriving, the browser deciding to do housekeeping. The median of three passes is used precisely so that one interrupted pass does not decide the result — but it can only defend against one.
• Heat. A run leaves the device warmer than it found it. A second run started immediately begins from a higher temperature and may throttle sooner. The throttling guide shows how to see that on the chart.
• Which cores do the work. The processor test runs one worker per core, but the operating system decides which core each worker lands on, and on a chip with big and little cores that decision changes the answer. The Core Structure panel shows the spread between workers for exactly this reason.
• Power state. Plugged in or not, battery saver on or off, screen brightness. Any of these can change the clock the processor is allowed to run at, silently, between runs.

How to compare honestly

Treat one run as one sample. If a number matters, run again and see whether it holds. Two runs that agree are worth more than one run that looks precise. The comparison tool on the results screen keeps your previous runs so they can sit side by side, and the difference between two runs of the same device is itself a measurement — of how repeatable that device is.

When two devices are being compared, give them the same conditions: both cool, both on the same power state, both with nothing else running. A comparison made under different conditions measures the conditions.

And read the 1% low before the average. It moves for the same reasons the average does, but it moves in the direction you would notice, which makes it the better guide to whether a difference between two runs is one you would feel.

Related

• What does the score actually mean?Is 1,795 a good CPU score for a phone?
• Why is my new phone slower than my old one?My new phone feels slower than my old phone. Why?
• What makes a device throttle?What does it mean when a phone or laptop throttles?
• Is my phone throttling?How do I know if my phone is throttling?
• What is 1% low FPS?What does 1% low FPS mean?
• How to test a phone in a shopCan I test how fast a phone is before buying it?