Thinking mode?

[lix8840557-art]

Hi Bowen, quick question in your reply above and I saw the wording "enable thinking" in the paper as well. How can you do that for Qwen3-4B-Instruct-2507 which is a non-thinking model

Originally posted by @lix8840557-art in #7

[bowen-upenn]

Sorry for the confusion. I verified against our code repository and here is a more detailed answer.

For SFT, we didn't prepare any thinking trajectories. The SFT stage applies the standard next-token cross-entropy objective to the target tokens, and does not use Qwen's thinking mode. We intentionally use Qwen3-4B-Instruct-2507 rather than Qwen3-4B-Thinking, as the Thinking variant has already undergone substantial post-training. In our experience, starting from a heavily post-trained reasoning model may not be ideal for a relatively small-scale RL experiment on a single dataset, although it could still be a viable choice that you can experiment with.

During the GRPO stage, we begin with the SFT cold started Qwen3-4B-Instruct-2507, and then simply use a system prompt to ask the model to put its reasoning inside and before giving the final answer. We then use a format reward to enforce this thinking structure. This is how we manually enable the thinking mode.