diff --git a/quantecon/README.md b/quantecon/README.md new file mode 100644 index 000000000..ea3305f6f --- /dev/null +++ b/quantecon/README.md @@ -0,0 +1,175 @@ +# QuantEcon fork of `mystmd` — maintenance guide + +This fork lets QuantEcon develop and use new `mystmd` features before they have been reviewed and merged by the upstream `jupyter-book/mystmd` team. The goal is to make it as easy as possible to open upstream PRs from feature branches, while also being able to run a combined build that includes all in-progress features. + +## How it works — the key idea + +``` +jupyter-book/mystmd:main ─────┐ + │ (sync periodically) │ (feature branches start here) + ▼ │ +QuantEcon/mystmd:main │ ← mirror + quantecon/ tooling + │ │ + │ feature/foo ◄─────┤ ← branched from upstream/main; PR target = upstream main + │ feature/bar ◄─────┘ ← branched from upstream/main; PR target = upstream main + ▼ (built by build.sh, which merges feature branches into quantecon) +QuantEcon/mystmd:quantecon ← combined build: main + all feature branches +``` + +**Feature branches serve two purposes simultaneously:** + +1. They are the source of upstream PRs — targeting `jupyter-book/mystmd:main`, showing only the diff for that one feature. They are branched from `upstream/main` so they don't carry the `quantecon/` folder. +2. They are consumed by `build.sh`, which merges them onto the throwaway `quantecon` branch (which inherits the `quantecon/` folder from `main`). + +This means **you never need to choose** between "make it easy to upstream" and "make it available to use now". The feature branch does both. + +## Branching model + +| Branch | Purpose | +|---|---| +| `main` | Mirrors `jupyter-book/mystmd:main` exactly. Synced via the GitHub "Sync fork" button or `git merge upstream/main`. **No direct commits.** | +| `feature/` | One branch per logical patch. **Branched from `upstream/main`** (not `main`), kept rebased on `upstream/main`. This is the branch you open the upstream PR from. | +| `quantecon` | Throwaway combined build. Rebuilt by `build.sh` as `main` + all active feature branches. **Never commit here directly — it is always discarded and regenerated.** | + +## One-time setup + +```bash +git remote add upstream https://github.com/jupyter-book/mystmd.git +``` + +Verify your remotes look like this: + +``` +origin https://github.com/QuantEcon/mystmd.git (fetch/push) +upstream https://github.com/jupyter-book/mystmd.git (fetch/push) +``` + +## Regular workflow + +### Sync `main` with upstream + +Either use the **"Sync fork"** button on the GitHub web UI (simplest), or locally: + +```bash +git fetch upstream +git checkout main +git merge upstream/main +git push origin main +``` + +After syncing, rebuild the `quantecon` branch (see below) so it includes the latest upstream changes. + +### Develop a new feature + +> **Important:** branch from `upstream/main`, **not** from `main`. This keeps the `quantecon/` folder out of your feature branch, so the upstream PR diff shows only your actual changes. + +```bash +git fetch upstream +git checkout -b feature/ upstream/main +# make your changes and commit them +git push origin feature/ +``` + +Then: +1. Add the branch name to `quantecon/features.txt` (on `main`, then commit and push) +2. Open a PR on GitHub: **base: `jupyter-book/mystmd:main`**, **compare: `QuantEcon/mystmd:feature/`** +3. Rebuild the `quantecon` branch so the feature is available immediately + +> **Why this works:** `build.sh` runs from `main` (which has the `quantecon/` folder), checks out the `quantecon` branch, and merges your feature branch into it. Because git merges *changes* (not full trees), the feature branch doesn't need to contain `quantecon/` — the folder comes from `main`, and your feature's changes are layered on top. + +### Migrating an existing feature branch to use upstream/main as base + +If you already have a feature branch that was created from `main` (and therefore includes the `quantecon/` folder commits), rebase it onto `upstream/main`: + +```bash +git fetch upstream +git checkout feature/ +git rebase --onto upstream/main main +git push --force-with-lease origin feature/ +``` + +This replays only your feature's commits on top of `upstream/main`, dropping the `quantecon/` folder commits from the branch's history. The upstream PR diff will then show only your actual changes. + +### Rebuild the `quantecon` branch + +```bash +./quantecon/build.sh # local only (safe, no push) +./quantecon/build.sh --push # build and push to origin/quantecon +``` + +The script: +1. Syncs local `main` from `origin/main` +2. Resets `quantecon` to `main` (discarding any previous build) +3. Merges each branch listed in `features.txt` in order (merge commits, not squash) +4. Patches `packages/mystmd/package.json` version to append `-qe` (e.g. `1.9.0-qe`). The `copy:version` build step propagates this to `version.ts` at build time, so `myst --version` identifies the QuantEcon build. +5. Optionally pushes to `origin/quantecon` + +Run this after every upstream sync, after updating any feature branch, or after adding/removing a feature branch from `features.txt`. + +### Update a feature branch (e.g. to address upstream reviewer feedback) + +```bash +git checkout feature/ +# make changes, amend or add commits +git push --force-with-lease origin feature/ +# the upstream PR updates automatically +./quantecon/build.sh --push # rebuild quantecon with the updated branch +``` + +### Keep a feature branch current with upstream + +If `upstream/main` has moved since you branched: + +```bash +git fetch upstream +git checkout feature/ +git rebase upstream/main +git push --force-with-lease origin feature/ +./quantecon/build.sh --push +``` + +## Resolving merge conflicts + +Conflicts are **always fixed on the feature branch**, never on `quantecon` (which is throwaway and rebuilt from scratch each time). + +`build.sh` will abort cleanly and tell you which branch caused the conflict. + +**Feature conflicts with upstream** (upstream changed code the feature touches): +```bash +git fetch upstream +git checkout feature/ +git rebase upstream/main # resolve conflicts here, then: git add . && git rebase --continue +git push --force-with-lease origin feature/ +./quantecon/build.sh --push +``` + +**Feature A conflicts with feature B** (two patches touch the same lines): +```bash +# Rebase the later branch on top of the earlier one to establish ordering +git checkout feature/ +git rebase feature/ +git push --force-with-lease origin feature/ +# Make sure features.txt lists feature/ before feature/ +./quantecon/build.sh --push +``` + +## When upstream merges a feature + +Once the upstream PR is merged into `jupyter-book/mystmd:main`: + +1. Sync `main` with upstream — it now contains the feature +2. Remove the branch from `features.txt` +3. Delete the feature branch +4. Rebuild `quantecon` — the feature is now in `main` itself, so nothing is lost + +```bash +git fetch upstream +git checkout main && git merge upstream/main && git push origin main +# edit features.txt to remove the branch +git branch -d feature/ && git push origin --delete feature/ +./quantecon/build.sh --push +``` + +## Active patches + +See [features.txt](features.txt) for the current list of carry-patches and links to their upstream PRs. diff --git a/quantecon/build.sh b/quantecon/build.sh new file mode 100644 index 000000000..7ebdb3a9d --- /dev/null +++ b/quantecon/build.sh @@ -0,0 +1,199 @@ +#!/usr/bin/env bash +# quantecon/build.sh +# +# Rebuilds the `quantecon` integration branch by merging all active feature +# branches (listed in quantecon/features.txt) on top of current `main`. +# +# Usage: +# ./quantecon/build.sh [--push] +# +# Options: +# --push Push the resulting branch to origin after a successful build. +# +# Conflict resolution: +# This script aborts cleanly on any merge conflict. Conflicts must be +# resolved on the feature branch itself (never on `quantecon`): +# +# Feature vs. main: +# git checkout && git rebase main +# # resolve conflicts, git add, git rebase --continue +# git push --force-with-lease origin +# ./quantecon/build.sh [--push] +# +# Feature A vs. feature B: +# git checkout && git rebase +# # resolve conflicts, update features.txt so branch-A appears before branch-B +# git push --force-with-lease origin +# ./quantecon/build.sh [--push] + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FEATURES_FILE="$SCRIPT_DIR/features.txt" +INTEGRATION_BRANCH="quantecon" +BASE_BRANCH="main" +PUSH=false + +# --------------------------------------------------------------------------- +# Parse args +# --------------------------------------------------------------------------- +for arg in "$@"; do + case "$arg" in + --push) PUSH=true ;; + *) echo "Unknown argument: $arg" >&2; exit 1 ;; + esac +done + +# --------------------------------------------------------------------------- +# Preflight checks +# --------------------------------------------------------------------------- +if ! git rev-parse --git-dir > /dev/null 2>&1; then + echo "ERROR: not inside a git repository." >&2 + exit 1 +fi + +if [[ -n "$(git status --porcelain)" ]]; then + echo "ERROR: working tree is dirty. Commit or stash changes before running." >&2 + exit 1 +fi + +# Read feature branches (strip comments and blank lines, extract first token) +mapfile -t FEATURES < <(awk '/^[[:space:]]*#/{next} /^[[:space:]]*$/{next} {print $1}' "$FEATURES_FILE") + +if [[ ${#FEATURES[@]} -eq 0 ]]; then + echo "No feature branches listed in $FEATURES_FILE. Nothing to do." + exit 0 +fi + +echo "==> Feature branches to merge:" +for f in "${FEATURES[@]}"; do + echo " $f" +done + +# --------------------------------------------------------------------------- +# Ensure local main is up to date +# --------------------------------------------------------------------------- +echo "" +echo "==> Fetching origin..." +git fetch origin + +echo "" +echo "==> Switching to $BASE_BRANCH and fast-forwarding from origin..." +git checkout "$BASE_BRANCH" +# main should be a pure upstream mirror — ff-only guards against accidental commits +if ! git merge --ff-only origin/"$BASE_BRANCH"; then + echo "ERROR: could not fast-forward $BASE_BRANCH from origin/$BASE_BRANCH." \ + "Sync main with upstream first (git merge upstream/main && git push origin main)." >&2 + exit 1 +fi + +BASE_SHA=$(git rev-parse "$BASE_BRANCH") +echo " Base commit: $BASE_SHA" + +# --------------------------------------------------------------------------- +# Verify all feature branches exist (locally or on origin) +# --------------------------------------------------------------------------- +echo "" +echo "==> Verifying feature branches exist..." +for branch in "${FEATURES[@]}"; do + if git rev-parse --verify "$branch" > /dev/null 2>&1; then + echo " $branch (local)" + elif git rev-parse --verify "origin/$branch" > /dev/null 2>&1; then + # Create a local tracking branch so 'git merge ' works + git branch --track "$branch" "origin/$branch" 2>/dev/null || git branch -f "$branch" "origin/$branch" + echo " $branch (fetched from origin)" + else + echo "ERROR: branch '$branch' not found locally or on origin." >&2 + exit 1 + fi +done + +# --------------------------------------------------------------------------- +# Rebuild the integration branch +# --------------------------------------------------------------------------- +echo "" +echo "==> Resetting '$INTEGRATION_BRANCH' to '$BASE_BRANCH'..." +git checkout -B "$INTEGRATION_BRANCH" "$BASE_BRANCH" + +MERGED=() +for branch in "${FEATURES[@]}"; do + echo "" + echo "==> Merging $branch..." + if git merge --no-ff "$branch" -m "chore: merge $branch into $INTEGRATION_BRANCH"; then + MERGED+=("$branch") + else + echo "" >&2 + echo "ERROR: merge conflict when merging '$branch'." >&2 + echo "" >&2 + echo "Conflicting files:" >&2 + git diff --name-only --diff-filter=U >&2 + echo "" >&2 + echo "Aborting. The '$INTEGRATION_BRANCH' branch has been reset; no changes were pushed." >&2 + git merge --abort + git checkout "$BASE_BRANCH" + git branch -D "$INTEGRATION_BRANCH" + echo "" >&2 + echo "How to fix:" >&2 + if [[ ${#MERGED[@]} -gt 0 ]]; then + echo " '$branch' conflicts with one of: ${MERGED[*]}" >&2 + echo " Rebase '$branch' on top of the conflicting branch:" >&2 + echo " git checkout $branch && git rebase " >&2 + echo " Then update features.txt so the earlier branch appears first." >&2 + else + echo " '$branch' conflicts with $BASE_BRANCH." >&2 + echo " Rebase it: git checkout $branch && git rebase $BASE_BRANCH" >&2 + fi + echo " Resolve conflicts, push the feature branch, then re-run this script." >&2 + exit 1 + fi +done + +# --------------------------------------------------------------------------- +# Patch version string with -qe suffix +# --------------------------------------------------------------------------- +# version.ts is generated/gitignored — patch packages/mystmd/package.json instead. +# The copy:version build step will propagate the version to version.ts at build time. +PACKAGE_JSON="packages/mystmd/package.json" +echo "" +echo "==> Patching version in $PACKAGE_JSON with '-qe' suffix..." + +CURRENT_VERSION=$(node -p "require('./$PACKAGE_JSON').version") +QE_VERSION="${CURRENT_VERSION}-qe" + +# Use node to patch the JSON safely (avoids sed quoting issues with JSON) +node -e " + const fs = require('fs'); + const pkg = JSON.parse(fs.readFileSync('$PACKAGE_JSON', 'utf8')); + pkg.version = '$QE_VERSION'; + fs.writeFileSync('$PACKAGE_JSON', JSON.stringify(pkg, null, 2) + '\n'); +" + +git add "$PACKAGE_JSON" +git commit -m "chore: set version to ${QE_VERSION} for QuantEcon build" +echo " Version set to ${QE_VERSION}" + +# --------------------------------------------------------------------------- +# Summary +# --------------------------------------------------------------------------- +echo "" +echo "==> Build successful." +echo " Branch '$INTEGRATION_BRANCH' contains $BASE_BRANCH + ${#MERGED[@]} feature branch(es):" +for f in "${MERGED[@]}"; do + echo " $f" +done +echo " Version: ${QE_VERSION}" + +# --------------------------------------------------------------------------- +# Optionally push +# --------------------------------------------------------------------------- +if $PUSH; then + echo "" + echo "==> Pushing '$INTEGRATION_BRANCH' to origin (force-with-lease)..." + git push --force-with-lease origin "$INTEGRATION_BRANCH" + echo " Done." +else + echo "" + echo "Tip: run with --push to push to origin/$INTEGRATION_BRANCH." +fi + +git checkout "$BASE_BRANCH" diff --git a/quantecon/features.txt b/quantecon/features.txt new file mode 100644 index 000000000..31e051c4e --- /dev/null +++ b/quantecon/features.txt @@ -0,0 +1,12 @@ +# Active QuantEcon feature branches +# One branch name per line. Lines starting with # are ignored. +# Order matters: branches are merged in sequence. +# If two branches conflict with each other, rebase the later one on top +# of the earlier one and update both entries accordingly. +# +# Format: +# [# optional description / upstream PR link] +# +# Example: +# feature/xref-improvements # https://github.com/jupyter-book/mystmd/pull/1 +