The Problem
Hi, thanks a lot for releasing HRM and the code; the results on ARC-AGI, Sudoku-Extreme, and Maze-Hard are very impressive. I have a question about the baseline evaluations in Figure 1 in the paper, in particular the CoT models: the CoT baselines (o3-mini-high, DeepSeek R1, Claude 3.7 8K) show very low accuracy, and 0% on Sudoku-Extreme and Maze-Hard, while HRM performs strongly on these tasks.
In Section 3.2 in the paper you mention that "for ARC-AGI, the scores of the CoT models are taken from the official leaderboard, while for Sudoku and Maze, the scores are obtained by evaluating through the corresponding API," and that the baselines use chain-of-thought prompting.
After reading the paper and going through the code, I have not been able to find details regarding the baselines.
The Questions
I am especially interested in the exact prompts that were used for each baseline model, including the full system and user messages, any few‑shot examples, and any task‑specific instructions, for all four benchmarks (ARC-AGI-1, ARC-AGI-2, Sudoku-Extreme, and Maze-Hard). It would also be very helpful to know the precise generation settings for these API calls, such as the temperature, top‑p, max_tokens, stop sequences, as well as how many samples were generated per puzzle and whether you used majority voting or any other aggregation strategy.
Thank you for your time and for sharing this work!
The Problem
Hi, thanks a lot for releasing HRM and the code; the results on ARC-AGI, Sudoku-Extreme, and Maze-Hard are very impressive. I have a question about the baseline evaluations in Figure 1 in the paper, in particular the CoT models: the CoT baselines (o3-mini-high, DeepSeek R1, Claude 3.7 8K) show very low accuracy, and 0% on Sudoku-Extreme and Maze-Hard, while HRM performs strongly on these tasks.
In Section 3.2 in the paper you mention that "for ARC-AGI, the scores of the CoT models are taken from the official leaderboard, while for Sudoku and Maze, the scores are obtained by evaluating through the corresponding API," and that the baselines use chain-of-thought prompting.
After reading the paper and going through the code, I have not been able to find details regarding the baselines.
The Questions
I am especially interested in the exact prompts that were used for each baseline model, including the full system and user messages, any few‑shot examples, and any task‑specific instructions, for all four benchmarks (ARC-AGI-1, ARC-AGI-2, Sudoku-Extreme, and Maze-Hard). It would also be very helpful to know the precise generation settings for these API calls, such as the temperature, top‑p, max_tokens, stop sequences, as well as how many samples were generated per puzzle and whether you used majority voting or any other aggregation strategy.
Thank you for your time and for sharing this work!