SimAgentsExperiment 03 / Published results
Extended Connections
Can a model find the hidden pattern?
Models try 940 word-grouping puzzles with up to four extra words added as distractions. Each puzzle allows one attempt to find four groups.
MEASUREPuzzle score (0–100)
DIRECTIONHigher is better
UNCERTAINTYNone published
The results at a glance
Puzzle score (0–100)
Each bar shows the average puzzle score, from 0 to 100.
1
98.12
97.73
97.44
97.45
96.26
95.57
95.28
94.49
94.310
94.011
93.812
93.612 of 126 matching models shown · 940 puzzlesSelect a model’s name to add it to your comparison.
All results
Download table ↓All 940 puzzles. Every model in this ranking attempted the same puzzles. Newest-100 results and incomplete tests appear separately below.
126 of 126 models and settings shown. These rankings apply to this test and publication date.
Inspect the newest 100 puzzles Same models, a separate set of puzzles +
These are the most recent 100 puzzles in the published test. Looking at newer puzzles helps assess whether earlier puzzles might have appeared in a model’s training material. These scores are kept separate from the 940-puzzle ranking above. A lower score on the newest puzzles does not by itself show that older puzzles appeared in training: the earliest puzzles were also easier.
| Rank | Model and settings | Puzzle score (0–100) |
|---|---|---|
| 1 | GPT-6 Astra (xhigh reasoning) | 98.7 |
| 2 | GPT-6 Astra (high reasoning) | 97.8 |
| 3 | Gemini 3.8 Flash (high reasoning) | 96.5 |
| 4 | Gemini 3.1 Pro Preview | 96.1 |
| 5 | GPT-5.5 (xhigh reasoning) | 96.1 |
| 6 | GPT-6.1 Sol (high reasoning) | 95.6 |
| 7 | GPT-5.5 (high reasoning) | 95.2 |
| 8 | Claude Opus 5 (xhigh reasoning) | 94.5 |
| 9 | GPT-5.6 Sol (xhigh reasoning) | 94.4 |
| 10 | Claude Fable 5 (high reasoning) | 93.6 |
| 11 | GPT-5.4 (xhigh reasoning) | 93.4 |
| 12 | GPT-5.6 Sol (high reasoning) | 92.7 |
| 13 | Gemini 3 Pro Preview | 92.3 |
| 14 | Gemini 3.7 Flash | 92.2 |
| 15 | Claude Opus 4.8 (high reasoning) | 92.1 |
| 16 | Kimi K3 | 92.1 |
| 17 | Claude Opus 5 (high reasoning) | 91.9 |
| 18 | Gemini 3.5 Flash | 90.8 |
| 19 | Claude Opus 4.8 (xhigh reasoning) | 90.6 |
| 20 | Claude Opus 5.5 (high reasoning) | 90.6 |
| 21 | Qwen 3.8 Max | 88.9 |
| 22 | GPT-6 Sol (high reasoning) | 88.8 |
| 23 | Claude Fable 5.1 (high reasoning) | 88.5 |
| 24 | Gemini 3.6 Flash | 88.5 |
| 25 | DeepSeek V4 Pro (high reasoning) | 88.4 |
| 26 | GPT-5.4 (high reasoning) | 88.4 |
| 27 | Claude Opus 4.6 (high reasoning) | 88.1 |
| 28 | GPT-5.5 (medium reasoning) | 87.3 |
| 29 | DeepSeek V4 Flash | 87.2 |
| 30 | DeepSeek V4.1 Flash (high reasoning) | 86.1 |
| 31 | Grok 4.20 Multi-Agent Exp Beta 0304 | 85.2 |
| 32 | Muse Spark 1.1 (high reasoning) | 84.8 |
| 33 | GPT-5.4 (medium reasoning) | 84.4 |
| 34 | Muse Spark 1.3 (high reasoning) | 83.6 |
| 35 | Claude Opus 5.5 (medium reasoning) | 83.4 |
| 36 | Kimi K2.6 | 80.2 |
| 37 | Claude Sonnet 5.5 (high reasoning) | 79.1 |
| 38 | Qwen 3.7 Max | 78.9 |
| 39 | GPT-5.2 (xhigh reasoning) | 78.6 |
| 40 | Claude Opus 4.6 Thinking 16K | 78.4 |
| 41 | GLM-5.1 | 77.6 |
| 42 | Grok 4.20 0309 (Reasoning) | 77.4 |
| 43 | Muse Spark 1.2 (high reasoning) | 77.2 |
| 44 | Grok 4.5 (high reasoning) | 77.2 |
| 45 | Grok 4.20 Reasoning Exp Beta 0304 | 77.1 |
| 46 | Grok 4.6 (xhigh reasoning) | 77.1 |
| 47 | Grok 4.1 Fast Reasoning | 75.2 |
| 48 | Grok 4.6 (high reasoning) | 74.6 |
| 49 | Claude Sonnet 4.6 (high reasoning) | 74.1 |
| 50 | GLM-5.2 (high reasoning) | 72.8 |
| 51 | Grok 4.7 (high reasoning) | 72.8 |
| 52 | GPT-5.6 Terra (high reasoning) | 72.4 |
| 53 | GLM-5.3 (high reasoning) | 72.2 |
| 54 | GPT-5.2 Pro | 71.8 |
| 55 | Qwen 3.7 Plus | 70.9 |
| 56 | Claude Sonnet 5 (high reasoning) | 69.9 |
| 57 | Claude Sonnet 4.6 Thinking 32K | 69.4 |
| 58 | Qwen 3.6 Max Preview | 69.4 |
| 59 | GLM-5 | 67.5 |
| 60 | GPT-5.2 (high reasoning) | 66.2 |
| 61 | GPT-5.6 Luna (high reasoning) | 66.1 |
| 62 | Gemma 4 31B Reasoning | 65.9 |
| 63 | GPT-6 Luna (high reasoning) | 65.8 |
| 64 | Claude Haiku 5.5 (high reasoning) | 64.9 |
| 65 | Tencent Hy4 Preview | 64.2 |
| 66 | Kimi K2.5 Thinking | 63.7 |
| 67 | MiniMax-M3 | 63.4 |
| 68 | GPT-5.4 Mini (xhigh reasoning) | 62.5 |
| 69 | Gemini 3 Flash Preview | 61.8 |
| 70 | Gemini 3.5 Flash-Lite (high reasoning) | 60.7 |
| 71 | DeepSeek V4 Pro Preview | 59.9 |
| 72 | Qwen 3.6 Plus | 57.1 |
| 73 | GPT-5.2 (medium reasoning) | 55.8 |
| 74 | Qwen3.8-27B | 52.0 |
| 75 | Claude Opus 4.5 Thinking 16K | 51.6 |
| 76 | Qwen3.5-397B-A17B | 47.1 |
| 77 | GPT-5.2 (low reasoning) | 46.4 |
| 78 | Grok 4.3 | 45.6 |
| 79 | Qwen3.5-122B-A10B | 45.4 |
| 80 | Claude Sonnet 4.6 Thinking 16K | 45.3 |
| 81 | Qwen3.5-27B | 44.6 |
| 82 | Claude Sonnet 4.5 (no reasoning) | 43.6 |
| 83 | Claude Sonnet 4.5 Thinking 16K | 43.0 |
| 84 | Claude Opus 4.5 (no reasoning) | 42.0 |
| 85 | Tencent Hy3 (high) | 40.6 |
| 86 | Claude Sonnet 4.6 (no reasoning) | 40.4 |
| 87 | Claude Opus 4.7 (high reasoning) | 39.3 |
| 88 | Claude 4.5 Haiku | 37.5 |
| 89 | Claude Opus 4.6 (no reasoning) | 37.2 |
| 90 | Qwen3.6-35B-A3B | 35.5 |
| 91 | Qwen 3.7 Flash | 35.4 |
| 92 | Step 3.7 Flash (high reasoning) | 34.2 |
| 93 | Qwen3 Max (2026-01-23) | 31.6 |
| 94 | Xiaomi MiMo V2.5 Pro | 31.4 |
| 95 | DeepSeek V3.2 | 29.7 |
| 96 | MiniMax-M2.7 | 28.6 |
| 97 | ByteDance Seed2.0 Pro | 28.0 |
| 98 | Mistral Large 4 (high) | 27.2 |
| 99 | Step 3.5 Flash | 22.8 |
| 100 | Xiaomi MiMo V2 Pro | 22.0 |
| 101 | Baidu Ernie 5.1 | 20.0 |
| 102 | GPT-5.4 (no reasoning) | 19.8 |
| 103 | GPT-5.5 (no reasoning) | 19.1 |
| 104 | Tencent Hy3 Preview | 18.9 |
| 105 | LongCat Flash Thinking | 17.3 |
| 106 | Arcee Trinity Large Thinking | 16.9 |
| 107 | MiniMax-M2.5 | 16.2 |
| 108 | GPT-5.2 (no reasoning) | 15.8 |
| 109 | Gemma 4 31B IT | 15.7 |
| 110 | Mistral Medium 3.5 (high) | 14.6 |
| 111 | MiniMax-M2.1 | 12.8 |
| 112 | Nemotron 3 Super | 12.4 |
| 113 | Claude Opus 4.7 (no reasoning) | 11.9 |
| 114 | MiniMax-M2 | 11.9 |
| 115 | Qwen 3 Max Thinking | 11.6 |
| 116 | Baidu Ernie 5.0 | 10.7 |
| 117 | Grok 4.20 0309 (Non-Reasoning) | 9.4 |
| 118 | Mistral Large 3 | 8.9 |
| 119 | Gemini 3.1 Flash-Lite Preview | 8.9 |
| 120 | Grok 4.1 Fast Non-Reasoning | 8.8 |
| 121 | Tencent Hy3 (non-thinking) | 8.2 |
| 122 | Llama 4 Maverick | 8.1 |
| 123 | Grok 4.20 Non-Reasoning Exp Beta 0304 | 7.7 |
| 124 | Mistral Medium 3.1 | 7.1 |
| 125 | DeepSeek V3.2 (no reasoning) | 6.8 |
| 126 | Ling 2.6 1T | 5.5 |
Older & incomplete tests 114 results, separate from the main ranking +
These models were tested on fewer than 940 puzzles. Their scores cover different sets of puzzles and should not be compared directly with the main ranking.
| Model and settings | Puzzle score (0–100) | Puzzles |
|---|---|---|
| ByteDance Seed2.1 Pro | 97.8 | 101 / 940 |
| GLM-5.2 (max reasoning) | 89.5 | 194 / 940 |
| Sherlock Think Alpha | 89.0 | 759 / 940 |
| Qwen 3.8 Max 0902 | 88.8 | 919 / 940 |
| Grok 4 Fast Reasoning | 88.0 | 759 / 940 |
| Inkling (high reasoning) | 88.0 | 136 / 940 |
| Grok 4 | 87.8 | 759 / 940 |
| Sonoma Sky Alpha | 86.5 | 759 / 940 |
| Xiaomi MiMo V2.6 Pro (thinking) | 84.0 | 27 / 940 |
| o3-pro (medium reasoning) | 81.7 | 759 / 940 |
| GPT-5 Pro | 77.0 | 759 / 940 |
| GLM-5.2 (max reasoning) | 75.9 | 211 / 940 |
| o1-pro (medium reasoning) | 74.6 | 651 / 940 |
| o3 (high reasoning) | 70.5 | 759 / 940 |
| GPT-5 (high reasoning) | 67.8 | 759 / 940 |
| o4-mini (high reasoning) | 64.0 | 759 / 940 |
| Ox Alpha (max reasoning) | 63.5 | 217 / 940 |
| o3 (medium reasoning) | 63.0 | 759 / 940 |
| GPT-5 (medium reasoning) | 61.8 | 759 / 940 |
| o1 (medium reasoning) | 60.1 | 651 / 940 |
| GPT-5.1 (high reasoning) | 59.7 | 759 / 940 |
| o4-mini (medium reasoning) | 58.4 | 651 / 940 |
| GPT-5 mini (medium reasoning) | 54.9 | 759 / 940 |
| GPT-5 (low reasoning) | 53.2 | 759 / 940 |
| GPT-5.1 (medium reasoning) | 51.2 | 759 / 940 |
| o3-mini (high reasoning) | 49.0 | 651 / 940 |
| GLM-4.7 | 47.2 | 767 / 940 |
| Claude Opus 4.1 Thinking 16K | 45.3 | 759 / 940 |
| Gemini 2.5 Pro | 44.9 | 759 / 940 |
| Kimi K2 Thinking 64K | 44.6 | 924 / 940 |
| DeepSeek V4 Flash (thinking) | 43.9 | 759 / 940 |
| o3-mini (medium reasoning) | 40.1 | 651 / 940 |
| Gemini 2.5 Pro Exp 03-25 | 40.1 | 651 / 940 |
| Qwen 3 235B A22B | 39.8 | 759 / 940 |
| Claude Opus 4 Thinking 16K | 35.2 | 759 / 940 |
| DeepSeek R1 05/28 | 33.6 | 759 / 940 |
| Qwen 3 235B A22B 25-07 Think | 31.6 | 759 / 940 |
| Gemini 2.5 Pro Preview 05-06 | 28.1 | 651 / 940 |
| Claude Sonnet 4 Thinking 16K | 26.4 | 759 / 940 |
| Claude Sonnet 4 Thinking 64K | 26.0 | 651 / 940 |
| DeepSeek R1 | 24.4 | 651 / 940 |
| GPT-OSS-120B | 24.1 | 759 / 940 |
| Claude Opus 4.1 (no reasoning) | 21.7 | 759 / 940 |
| Qwen 3 30B A3B | 21.2 | 759 / 940 |
| Qwen 3 32B | 21.0 | 759 / 940 |
| Qwen 3 30B A3B 25-07 Thinking | 20.9 | 759 / 940 |
| Claude 3.7 Sonnet Thinking 16K | 19.7 | 651 / 940 |
| Claude Opus 4 (no reasoning) | 19.7 | 759 / 940 |
| GPT-4.5 Preview | 19.2 | 651 / 940 |
| Qwen 3 Next 80B A3B Thinking | 18.5 | 759 / 940 |
| Qwen QwQ-32B 16K | 17.8 | 651 / 940 |
| Grok 3 Mini Beta (high) | 17.2 | 759 / 940 |
| GLM-4.5 | 17.0 | 759 / 940 |
| o1-mini | 14.1 | 651 / 940 |
| Grok 3 Mini Beta (low) | 14.0 | 651 / 940 |
| GPT-5 (minimal reasoning) | 13.9 | 759 / 940 |
| Claude Sonnet 4 (no reasoning) | 13.7 | 759 / 940 |
| Claude Opus 4.6 Thinking 32K | 13.6 | 98 / 940 |
| Cohere Command A Reasoning | 13.3 | 759 / 940 |
| Gemini 2.5 Flash | 12.8 | 759 / 940 |
| Grok 4 Fast Non-Reasoning | 12.5 | 759 / 940 |
| GLM-4.6 | 12.5 | 759 / 940 |
| Sherlock Dash Alpha | 12.2 | 759 / 940 |
| Quasar Alpha | 12.1 | 651 / 940 |
| Gemini 2.0 Flash Think Exp 01-21 | 12.0 | 649 / 940 |
| Cohere Command A+ | 12.0 | 898 / 940 |
| GPT-4o Mar 2025 | 11.8 | 759 / 940 |
| Qwen 3 Max Preview | 11.6 | 759 / 940 |
| Gemini 2.0 Pro Exp 02-05 | 11.2 | 651 / 940 |
| Kimi K2-0905 | 11.0 | 759 / 940 |
| GPT-4.1 | 10.8 | 759 / 940 |
| Sonoma Dusk Alpha | 10.6 | 759 / 940 |
| MiniMax-M1 | 10.5 | 688 / 940 |
| GPT-4o Feb 2025 | 10.3 | 651 / 940 |
| Polaris Alpha | 10.2 | 759 / 940 |
| DeepSeek V3.1 Non-Think | 10.1 | 759 / 940 |
| GPT-5.1 (no reasoning) | 10.0 | 759 / 940 |
| Grok 3 Beta (no reasoning) | 9.0 | 759 / 940 |
| Kimi K2 | 8.9 | 759 / 940 |
| Claude 3.7 Sonnet | 8.7 | 651 / 940 |
| Gemini 1.5 Pro (Sept) | 8.7 | 601 / 940 |
| Qwen 3 235B A22B 25-07 Instruct | 8.6 | 759 / 940 |
| GPT-4o 2024-11-20 | 8.6 | 601 / 940 |
| Grok 2 12-12 | 8.5 | 651 / 940 |
| Gemini 2.0 Flash | 8.3 | 651 / 940 |
| Claude 3 Opus | 8.2 | 650 / 940 |
| Claude 3.5 Sonnet 2024-10-22 | 8.1 | 651 / 940 |
| GPT-4o 2024-08-06 | 7.6 | 601 / 940 |
| Llama 4 Scout | 7.6 | 759 / 940 |
| Qwen 2.5 Max | 7.6 | 651 / 940 |
| DeepSeek V3-0324 | 7.4 | 759 / 940 |
| Llama 3.1 405B | 6.7 | 651 / 940 |
| Baidu Ernie 4.5 300B A47B | 6.5 | 759 / 940 |
| DeepSeek V4 Flash | 6.2 | 651 / 940 |
| GPT-4.1 mini | 6.1 | 759 / 940 |
| Llama 3.3 70B | 5.9 | 651 / 940 |
| MiniMax-Text-01 | 5.9 | 759 / 940 |
| Mistral Medium 3.1 | 5.5 | 759 / 940 |
| LongCat Flash | 5.2 | 660 / 940 |
| Cohere Command A | 5.0 | 759 / 940 |
| Mistral Large 3 | 4.8 | 759 / 940 |
| Mistral Small 3.2 | 4.3 | 759 / 940 |
| Gemma 2 27B | 4.3 | 651 / 940 |
| Mistral Small 3.1 | 4.2 | 651 / 940 |
| Gemma 3 27B | 4.2 | 759 / 940 |
| Amazon Nova Pro | 4.1 | 759 / 940 |
| Qwen 2.5 72B | 4.0 | 759 / 940 |
| Claude 3.5 Haiku | 3.8 | 759 / 940 |
| Microsoft Phi-4 | 3.4 | 759 / 940 |
| GPT-4o mini | 3.4 | 759 / 940 |
| Mistral Small 3 | 3.0 | 601 / 940 |
| GPT-4.1 nano | 2.8 | 759 / 940 |
| GLM4-32B-0414 | 2.7 | 759 / 940 |
| Claude 3 Haiku | 0.7 | 601 / 940 |
Understanding the results
How the scores work.
Finding more of a puzzle’s four groups earns progressively more credit: 0, 6.25, 25, 56.25, or 100 points. The score is the average across puzzles. A score of 100 means every puzzle was fully solved.
For each puzzle, the fraction of groups solved is squared and multiplied by 100. No uncertainty ranges were published for these results.
Puzzles a model refused to answer, or could not answer because they were blocked, receive zero points.
The main ranking includes only models tested on all 940 puzzles. Results for the newest 100 puzzles and for incomplete tests are shown separately.
Results as of 2026-10-08. View the original publication ↗