feat(tutorial): add a CPU-first RL walkthrough

[geyanqi]

Motivation

Molt is already designed as a lightweight, hackable RL framework that makes
experimentation and modification easier than larger training stacks. However,
its production RL path still necessarily combines the learning algorithm with
Ray, vLLM, FSDP, CUDA, and distributed model execution, so inspecting or
changing the core reward-to-update logic still requires access to a GPU
environment.

This PR takes Molt's lightweight philosophy one step further by adding a
single-process CPU learning path. It lets contributors understand, debug, and
validate lightweight algorithm or data-flow changes locally, then transfer the
same changes to the GPU/distributed production path with a smaller and clearer
validation gap.

What this adds

• A linear walkthrough from dataset row to rollout, reward, experience, loss,
  backward, and optimizer update.
• A real CPU Embedding -> Linear policy and local generation transport.
• Pinned CPU-only tutorial dependencies.
• English and Chinese design overviews and detailed walkthroughs.

Scope

The tutorial directly reuses Molt's dataset, agent, environment, trajectory,
experience, advantage, replay-buffer, and loss implementations.

Only Ray scheduling, vLLM serving, AutoModel/FSDP/CUDA execution, and
distributed weight synchronization are replaced.

This is a learning and debugging harness, not a production CPU backend.

Validation

• Pre-commit passes.
• Compileall passes.
• Cached offline end-to-end execution passes without model weights.
• CPU-safe unit tests: 124 passed, 1 skipped.
• Existing production source and tests are unchanged.

Follow-up

A companion skill is planned to help users and coding agents prototype against
this CPU path and transfer changes to the GPU/distributed implementation.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Copilot]

Pull request overview

This PR adds a CPU-only, single-process RL tutorial path under examples/tutorial/ to make Molt’s reward-to-update flow easier to understand and debug locally without Ray/vLLM/FSDP/CUDA.

Changes:

• Added an end-to-end CPU RL walkthrough script that reuses Molt’s real dataset/rollout/experience/loss code while stubbing Ray + vLLM boundaries.
• Added English and Chinese tutorial documentation describing scope, source correspondence, and a suggested debugging/breakpoint order.
• Added a pinned CPU-only requirements file for running the tutorial in an isolated environment.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
examples/tutorial/WALKTHROUGH.md	English detailed walkthrough of the CPU learning path, scope, and execution chain.
examples/tutorial/WALKTHROUGH_CN.md	Chinese version of the detailed walkthrough.
examples/tutorial/requirements-cpu.txt	Pinned CPU-only dependency set for the tutorial environment.
examples/tutorial/README.md	English design/scope overview for the CPU-first tutorial directory.
examples/tutorial/README_CN.md	Chinese design/scope overview.
examples/tutorial/cpu_rl_walkthrough.py	Linear CPU-only executable walkthrough that stubs Ray/vLLM and runs real Molt RL components end-to-end.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chtruong814]

/claude review

[claude]

LGTM

Light review of the new examples/tutorial/ CPU RL walkthrough (6 new files, no production-code changes). Verified the Python entry point against current Molt source:

• _collect_prompt_batch, SamplesGenerator._filter_group/init, RemoteExperienceMaker.init/build_experiences, StepEnvRunner.execute, PromptDataset, NaiveReplayBuffer, balance_experiences, get_tokenizer, load_agent_runner, FixedKLController -- all signatures and call conventions match.
• PolicyLoss.forward returns exactly the 6-tuple the script unpacks; compute_approx_kl / agg_loss / log_probs_from_logits args match.
• The ray.get=identity and async_run_method_batch stubs line up with how RemoteExperienceMaker dispatches (empty_cache only fires under colocate_fsdp_models, disabled here).
• TinyVllmEngine.generate matches the (request_output, off_policy_len) / outputs contract in StepEnvRunner.
• Doc-referenced symbols (PolicyTrainer.training_step, FsdpStrategy.backward/optimizer_step, compute_advantages_and_returns, reinforce_baseline) all exist.

The script carries its own end-to-end assertions (rewards, shape alignment, KL-stub guards, before/after log-prob margin), so coverage for the changed code is inherent. No critical bugs, typos, or inaccurate docs found.