Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 175 additions & 0 deletions quantecon/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
# QuantEcon fork of `mystmd` — maintenance guide

This fork lets QuantEcon develop and use new `mystmd` features before they have been reviewed and merged by the upstream `jupyter-book/mystmd` team. The goal is to make it as easy as possible to open upstream PRs from feature branches, while also being able to run a combined build that includes all in-progress features.

## How it works — the key idea

```
jupyter-book/mystmd:main ─────┐
│ (sync periodically) │ (feature branches start here)
▼ │
QuantEcon/mystmd:main │ ← mirror + quantecon/ tooling
│ │
│ feature/foo ◄─────┤ ← branched from upstream/main; PR target = upstream main
│ feature/bar ◄─────┘ ← branched from upstream/main; PR target = upstream main
▼ (built by build.sh, which merges feature branches into quantecon)
QuantEcon/mystmd:quantecon ← combined build: main + all feature branches
```

**Feature branches serve two purposes simultaneously:**

1. They are the source of upstream PRs — targeting `jupyter-book/mystmd:main`, showing only the diff for that one feature. They are branched from `upstream/main` so they don't carry the `quantecon/` folder.
2. They are consumed by `build.sh`, which merges them onto the throwaway `quantecon` branch (which inherits the `quantecon/` folder from `main`).

This means **you never need to choose** between "make it easy to upstream" and "make it available to use now". The feature branch does both.

## Branching model

| Branch | Purpose |
|---|---|
| `main` | Mirrors `jupyter-book/mystmd:main` exactly. Synced via the GitHub "Sync fork" button or `git merge upstream/main`. **No direct commits.** |
| `feature/<name>` | One branch per logical patch. **Branched from `upstream/main`** (not `main`), kept rebased on `upstream/main`. This is the branch you open the upstream PR from. |
| `quantecon` | Throwaway combined build. Rebuilt by `build.sh` as `main` + all active feature branches. **Never commit here directly — it is always discarded and regenerated.** |

## One-time setup

```bash
git remote add upstream https://github.com/jupyter-book/mystmd.git
```

Verify your remotes look like this:

```
origin https://github.com/QuantEcon/mystmd.git (fetch/push)
upstream https://github.com/jupyter-book/mystmd.git (fetch/push)
```

## Regular workflow

### Sync `main` with upstream

Either use the **"Sync fork"** button on the GitHub web UI (simplest), or locally:

```bash
git fetch upstream
git checkout main
git merge upstream/main
git push origin main
```

After syncing, rebuild the `quantecon` branch (see below) so it includes the latest upstream changes.

### Develop a new feature

> **Important:** branch from `upstream/main`, **not** from `main`. This keeps the `quantecon/` folder out of your feature branch, so the upstream PR diff shows only your actual changes.

```bash
git fetch upstream
git checkout -b feature/<name> upstream/main
# make your changes and commit them
git push origin feature/<name>
```

Then:
1. Add the branch name to `quantecon/features.txt` (on `main`, then commit and push)
2. Open a PR on GitHub: **base: `jupyter-book/mystmd:main`**, **compare: `QuantEcon/mystmd:feature/<name>`**
3. Rebuild the `quantecon` branch so the feature is available immediately

> **Why this works:** `build.sh` runs from `main` (which has the `quantecon/` folder), checks out the `quantecon` branch, and merges your feature branch into it. Because git merges *changes* (not full trees), the feature branch doesn't need to contain `quantecon/` — the folder comes from `main`, and your feature's changes are layered on top.

### Migrating an existing feature branch to use upstream/main as base

If you already have a feature branch that was created from `main` (and therefore includes the `quantecon/` folder commits), rebase it onto `upstream/main`:

```bash
git fetch upstream
git checkout feature/<name>
git rebase --onto upstream/main main
git push --force-with-lease origin feature/<name>
```

This replays only your feature's commits on top of `upstream/main`, dropping the `quantecon/` folder commits from the branch's history. The upstream PR diff will then show only your actual changes.

### Rebuild the `quantecon` branch

```bash
./quantecon/build.sh # local only (safe, no push)
./quantecon/build.sh --push # build and push to origin/quantecon
```

The script:
1. Syncs local `main` from `origin/main`
2. Resets `quantecon` to `main` (discarding any previous build)
3. Merges each branch listed in `features.txt` in order (merge commits, not squash)
4. Patches `packages/mystmd/package.json` version to append `-qe` (e.g. `1.9.0-qe`). The `copy:version` build step propagates this to `version.ts` at build time, so `myst --version` identifies the QuantEcon build.
5. Optionally pushes to `origin/quantecon`

Run this after every upstream sync, after updating any feature branch, or after adding/removing a feature branch from `features.txt`.

### Update a feature branch (e.g. to address upstream reviewer feedback)

```bash
git checkout feature/<name>
# make changes, amend or add commits
git push --force-with-lease origin feature/<name>
# the upstream PR updates automatically
./quantecon/build.sh --push # rebuild quantecon with the updated branch
```

### Keep a feature branch current with upstream

If `upstream/main` has moved since you branched:

```bash
git fetch upstream
git checkout feature/<name>
git rebase upstream/main
git push --force-with-lease origin feature/<name>
./quantecon/build.sh --push
```

## Resolving merge conflicts

Conflicts are **always fixed on the feature branch**, never on `quantecon` (which is throwaway and rebuilt from scratch each time).

`build.sh` will abort cleanly and tell you which branch caused the conflict.

**Feature conflicts with upstream** (upstream changed code the feature touches):
```bash
git fetch upstream
git checkout feature/<name>
git rebase upstream/main # resolve conflicts here, then: git add . && git rebase --continue
git push --force-with-lease origin feature/<name>
./quantecon/build.sh --push
```

**Feature A conflicts with feature B** (two patches touch the same lines):
```bash
# Rebase the later branch on top of the earlier one to establish ordering
git checkout feature/<name-b>
git rebase feature/<name-a>
git push --force-with-lease origin feature/<name-b>
# Make sure features.txt lists feature/<name-a> before feature/<name-b>
./quantecon/build.sh --push
```

## When upstream merges a feature

Once the upstream PR is merged into `jupyter-book/mystmd:main`:

1. Sync `main` with upstream — it now contains the feature
2. Remove the branch from `features.txt`
3. Delete the feature branch
4. Rebuild `quantecon` — the feature is now in `main` itself, so nothing is lost

```bash
git fetch upstream
git checkout main && git merge upstream/main && git push origin main
# edit features.txt to remove the branch
git branch -d feature/<name> && git push origin --delete feature/<name>
./quantecon/build.sh --push
```

## Active patches

See [features.txt](features.txt) for the current list of carry-patches and links to their upstream PRs.
199 changes: 199 additions & 0 deletions quantecon/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
#!/usr/bin/env bash
# quantecon/build.sh
#
# Rebuilds the `quantecon` integration branch by merging all active feature
# branches (listed in quantecon/features.txt) on top of current `main`.
#
# Usage:
# ./quantecon/build.sh [--push]
#
# Options:
# --push Push the resulting branch to origin after a successful build.
#
# Conflict resolution:
# This script aborts cleanly on any merge conflict. Conflicts must be
# resolved on the feature branch itself (never on `quantecon`):
#
# Feature vs. main:
# git checkout <branch> && git rebase main
# # resolve conflicts, git add, git rebase --continue
# git push --force-with-lease origin <branch>
# ./quantecon/build.sh [--push]
#
# Feature A vs. feature B:
# git checkout <branch-B> && git rebase <branch-A>
# # resolve conflicts, update features.txt so branch-A appears before branch-B
# git push --force-with-lease origin <branch-B>
# ./quantecon/build.sh [--push]

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
FEATURES_FILE="$SCRIPT_DIR/features.txt"
INTEGRATION_BRANCH="quantecon"
BASE_BRANCH="main"
PUSH=false

# ---------------------------------------------------------------------------
# Parse args
# ---------------------------------------------------------------------------
for arg in "$@"; do
case "$arg" in
--push) PUSH=true ;;
*) echo "Unknown argument: $arg" >&2; exit 1 ;;
esac
done

# ---------------------------------------------------------------------------
# Preflight checks
# ---------------------------------------------------------------------------
if ! git rev-parse --git-dir > /dev/null 2>&1; then
echo "ERROR: not inside a git repository." >&2
exit 1
fi

if [[ -n "$(git status --porcelain)" ]]; then
echo "ERROR: working tree is dirty. Commit or stash changes before running." >&2
exit 1
fi

# Read feature branches (strip comments and blank lines, extract first token)
mapfile -t FEATURES < <(awk '/^[[:space:]]*#/{next} /^[[:space:]]*$/{next} {print $1}' "$FEATURES_FILE")

if [[ ${#FEATURES[@]} -eq 0 ]]; then
echo "No feature branches listed in $FEATURES_FILE. Nothing to do."
exit 0
fi

echo "==> Feature branches to merge:"
for f in "${FEATURES[@]}"; do
echo " $f"
done

# ---------------------------------------------------------------------------
# Ensure local main is up to date
# ---------------------------------------------------------------------------
echo ""
echo "==> Fetching origin..."
git fetch origin

echo ""
echo "==> Switching to $BASE_BRANCH and fast-forwarding from origin..."
git checkout "$BASE_BRANCH"
# main should be a pure upstream mirror — ff-only guards against accidental commits
if ! git merge --ff-only origin/"$BASE_BRANCH"; then
echo "ERROR: could not fast-forward $BASE_BRANCH from origin/$BASE_BRANCH." \
"Sync main with upstream first (git merge upstream/main && git push origin main)." >&2
exit 1
fi

BASE_SHA=$(git rev-parse "$BASE_BRANCH")
echo " Base commit: $BASE_SHA"

# ---------------------------------------------------------------------------
# Verify all feature branches exist (locally or on origin)
# ---------------------------------------------------------------------------
echo ""
echo "==> Verifying feature branches exist..."
for branch in "${FEATURES[@]}"; do
if git rev-parse --verify "$branch" > /dev/null 2>&1; then
echo " $branch (local)"
elif git rev-parse --verify "origin/$branch" > /dev/null 2>&1; then
# Create a local tracking branch so 'git merge <branch>' works
git branch --track "$branch" "origin/$branch" 2>/dev/null || git branch -f "$branch" "origin/$branch"
echo " $branch (fetched from origin)"
else
echo "ERROR: branch '$branch' not found locally or on origin." >&2
exit 1
fi
done

# ---------------------------------------------------------------------------
# Rebuild the integration branch
# ---------------------------------------------------------------------------
echo ""
echo "==> Resetting '$INTEGRATION_BRANCH' to '$BASE_BRANCH'..."
git checkout -B "$INTEGRATION_BRANCH" "$BASE_BRANCH"

MERGED=()
for branch in "${FEATURES[@]}"; do
echo ""
echo "==> Merging $branch..."
if git merge --no-ff "$branch" -m "chore: merge $branch into $INTEGRATION_BRANCH"; then
Comment thread
mmcky marked this conversation as resolved.
MERGED+=("$branch")
else
echo "" >&2
echo "ERROR: merge conflict when merging '$branch'." >&2
echo "" >&2
echo "Conflicting files:" >&2
git diff --name-only --diff-filter=U >&2
echo "" >&2
echo "Aborting. The '$INTEGRATION_BRANCH' branch has been reset; no changes were pushed." >&2
git merge --abort
git checkout "$BASE_BRANCH"
git branch -D "$INTEGRATION_BRANCH"
echo "" >&2
echo "How to fix:" >&2
if [[ ${#MERGED[@]} -gt 0 ]]; then
echo " '$branch' conflicts with one of: ${MERGED[*]}" >&2
echo " Rebase '$branch' on top of the conflicting branch:" >&2
echo " git checkout $branch && git rebase <earlier-branch>" >&2
echo " Then update features.txt so the earlier branch appears first." >&2
else
echo " '$branch' conflicts with $BASE_BRANCH." >&2
echo " Rebase it: git checkout $branch && git rebase $BASE_BRANCH" >&2
fi
echo " Resolve conflicts, push the feature branch, then re-run this script." >&2
exit 1
fi
done

# ---------------------------------------------------------------------------
# Patch version string with -qe suffix
# ---------------------------------------------------------------------------
# version.ts is generated/gitignored — patch packages/mystmd/package.json instead.
# The copy:version build step will propagate the version to version.ts at build time.
PACKAGE_JSON="packages/mystmd/package.json"
echo ""
echo "==> Patching version in $PACKAGE_JSON with '-qe' suffix..."

CURRENT_VERSION=$(node -p "require('./$PACKAGE_JSON').version")
QE_VERSION="${CURRENT_VERSION}-qe"

# Use node to patch the JSON safely (avoids sed quoting issues with JSON)
node -e "
const fs = require('fs');
const pkg = JSON.parse(fs.readFileSync('$PACKAGE_JSON', 'utf8'));
pkg.version = '$QE_VERSION';
fs.writeFileSync('$PACKAGE_JSON', JSON.stringify(pkg, null, 2) + '\n');
"

git add "$PACKAGE_JSON"
git commit -m "chore: set version to ${QE_VERSION} for QuantEcon build"
echo " Version set to ${QE_VERSION}"

# ---------------------------------------------------------------------------
# Summary
# ---------------------------------------------------------------------------
echo ""
echo "==> Build successful."
echo " Branch '$INTEGRATION_BRANCH' contains $BASE_BRANCH + ${#MERGED[@]} feature branch(es):"
for f in "${MERGED[@]}"; do
echo " $f"
done
echo " Version: ${QE_VERSION}"

# ---------------------------------------------------------------------------
# Optionally push
# ---------------------------------------------------------------------------
if $PUSH; then
echo ""
echo "==> Pushing '$INTEGRATION_BRANCH' to origin (force-with-lease)..."
git push --force-with-lease origin "$INTEGRATION_BRANCH"
echo " Done."
else
echo ""
echo "Tip: run with --push to push to origin/$INTEGRATION_BRANCH."
fi

git checkout "$BASE_BRANCH"
12 changes: 12 additions & 0 deletions quantecon/features.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Active QuantEcon feature branches
# One branch name per line. Lines starting with # are ignored.
# Order matters: branches are merged in sequence.
# If two branches conflict with each other, rebase the later one on top
# of the earlier one and update both entries accordingly.
#
# Format:
# <branch-name> [# optional description / upstream PR link]
#
# Example:
# feature/xref-improvements # https://github.com/jupyter-book/mystmd/pull/1

Loading