Guides
Are online brain tests accurate?
Some are, once averaged. Simple reaction time reached a reliability of 0.83 on personal phones over three weeks (Van Patten et al., 2021), and averaging brief smartphone tests gave reliabilities of 0.97 or more (Sliwinski et al., 2018). Others, like the Stroop and flanker costs, stay weak even in a lab (Hedge et al., 2018).
What reliability means, in one picture
Imagine 50 people who take the same test today and again next week. If the people who scored high today also score high next week, and the low ones stay low, the test is reliable. Researchers summarise this with a test retest coefficient, often an intraclass correlation, from 0 for pure noise to 1 for a perfectly stable ranking.
Reliability is not accuracy in the everyday sense. A test can be very reliable and measure something narrow. But a test that is not reliable cannot tell you much about one person, because your score would change if you took it again tomorrow.
Reliability by task
The figures below come from published test retest studies. They are not all from the same format as the games on this site, which is why the parent task is named.
| Task | Reliability | Conditions |
|---|---|---|
| Simple reaction time, lab | 0.83 | 48 adults, three weekly sessions (Woods et al., 2015b) |
| Simple reaction time, personal phones | 0.83 | 55 adults, three weekly sessions (Van Patten et al., 2021) |
| Go/no-go commission errors | 0.76 | Lab, two sessions (Hedge et al., 2018) |
| Impulse control on phones | 0.68 to 0.80 | Times, then composite score (Van Patten et al., 2021) |
| Working memory on phones | 0.88 | Composite score, three weeks (Van Patten et al., 2021) |
| Trail Making A, computerized | 0.87 | 50 adults, three sessions (Woods et al., 2015c) |
| Corsi block span, computerized | 0.85 | Young adults, 20 minutes apart (Morawietz et al., 2024) |
| Stroop cost | 0.60 and 0.66 | Two lab studies (Hedge et al., 2018) |
| Flanker cost | 0.40 and 0.57 | Two lab studies (Hedge et al., 2018) |
The reliability paradox
The Stroop and flanker effects are among the most robust results in psychology. Almost everyone shows them. That is exactly the problem for measuring one person. When everyone has nearly the same effect, the differences between people are small compared with day to day noise, and the ranking does not hold from one session to the next.
Hedge, Powell and Sumner tested seven classic tasks and found reliabilities from 0 to 0.82, surprisingly low for tasks so widely used, and showed it came from low variation between people rather than from bad measurement (Hedge et al., 2018). The lesson for a scoring app is simple. Score what stays stable, keep the rest as games.
Your device changes your score
Online tests run on hardware nobody controls. Conventional touchscreens add 50 to 200 ms of latency (Van Patten et al., 2021). In 59,587 visitors to TestMyBrain.org, people on mobile devices, Android phones in particular, were slower on reaction time tests than people on computers, even after accounting for age, gender and education (Passell et al., 2021). A score is only comparable with scores taken on similar devices.
Practice inflates the first sessions
Scores go up when you repeat a test (Scharfen et al., 2018a). Across 68 studies of working memory tests, the gain from the first to the second session was about a quarter of a standard deviation, and scores kept rising up to the fourth session (Scharfen et al., 2018b). A big jump after your first games is mostly the test becoming familiar.
Averaging helps a lot. In 219 adults who took brief tests on their smartphones many times, the average scores reached a reliability of 0.97 or more, comparable to lab assessments (Sliwinski et al., 2018).
How Headgame takes all this into account
The card only scores tasks whose own reliability, or that of a close parent task, is known to be good, simple reaction time, the Schulte grid, sequence memory, the chimp test and go/no-go. Stroop and flanker stay free games. Each stat averages your two best sessions of the week, never a single morning, and the app corrects for the type of screen. The full method is on the science page.
Sources
- Van Patten R., Iverson G.L., Muzeau M.A., VanRavenhorst-Bell H.A. (2021). Test-retest reliability and reliable change estimates for four mobile cognitive tests administered virtually in community-dwelling adults. Frontiers in Psychology 12, 734947. PMC8566921. doi.org/10.3389/fpsyg.2021.734947
- Sliwinski M.J., Mogle J.A., Hyun J. et al. (2018). Reliability and validity of ambulatory cognitive assessments. Assessment 25(1), 14 to 30. PMC5690878. doi.org/10.1177/1073191116643164
- Hedge C., Powell G., Sumner P. (2018). The reliability paradox. Behavior Research Methods 50(3), 1166 to 1186. PMC5990556. doi.org/10.3758/s13428-017-0935-1
- Woods D.L., Wyma J.M., Yund E.W., Herron T.J. (2015). The effects of repeated testing, simulated malingering, and traumatic brain injury on high-precision measures of simple visual reaction time. Frontiers in Human Neuroscience 9, 540. PMC4637414. doi.org/10.3389/fnhum.2015.00540
- Woods D.L., Wyma J.M., Herron T.J., Yund E.W. (2015). The effects of aging, malingering, and traumatic brain injury on computerized Trail-Making Test performance. PLoS One 10(6), e0124345. PMC4465490. doi.org/10.1371/journal.pone.0124345
- Morawietz C. et al. (2024). Consistency of spatial ability performance in children, adolescents, and young adults. Frontiers in Psychology 15, 1365941. PMC10938598. doi.org/10.3389/fpsyg.2024.1365941
- Passell E., Strong R.W., Rutter L.A. et al. (2021). Cognitive test scores vary with choice of personal digital device. Behavior Research Methods 53(6), 2544 to 2557. PMC8568735. doi.org/10.3758/s13428-021-01597-3
- Scharfen J., Peters J.M., Holling H. (2018). Retest effects in cognitive ability tests, a meta-analysis. Intelligence 67, 44 to 66. doi.org/10.1016/j.intell.2018.01.003
- Scharfen J., Jansen K., Holling H. (2018). Retest effects in working memory capacity tests, a meta-analysis. Psychonomic Bulletin and Review 25, 2175 to 2199. doi.org/10.3758/s13423-018-1461-6
Headgame
Your full card, every Sunday.
Three 20 second games each morning, phone down sessions, and a card with six stats revealed every Sunday. Coming soon to the App Store, Google Play next.
Join the waitlist