ml/64-mingw-sparse

[matthew-levan]

Windows loom: demand paging and sparseness

Two branches, fourteen commits, ~1540 lines (about 1170 of them new Windows-only files). Branched from dbe52c39da on ml/64.

• ml/64-mingw-demand — 4 commits. Demand paging on Windows.
• ml/64-mingw-sparse — 10 more commits on top. A sparse loom, plus the migration path.

Why this was one piece of work

Two independent features that needed the same foundation:

• Demand paging. At boot, map image.bin into the loom instead of preading every page of it. Saves boot time and RAM proportional to snapshot size. Windows had it forced off unconditionally, with no flag to turn it back on.
• A sparse loom. Stop charging commit for the whole loom up front. This is what let a 16GB loom boot on a 2GB VM at all — it was failing before the demand-paging work could even be exercised.

Both need the loom to stop being one flat committed anonymous mapping.

The three POSIX assumptions Windows breaks

1. mmap(MAP_FIXED) replaces part of an existing mapping, atomically. MapViewOfFileEx fails if the address is occupied, and UnmapViewOfFile only accepts a whole view's base — so a hole cannot be punched in the loom to drop the image into. This blocked demand paging.

2. Overcommit. A Windows pagefile-backed section is charged against RAM plus the paging file when it is created, not when touched. This blocked booting entirely on a small box.

3. Uniform page protection. VirtualProtect cannot span separate mappings, cannot touch reserved pages, and rejects PAGE_READWRITE on a copy-on-write view.

Placeholder reservations (Windows 10 1803+) answer (1): reserve a range, split it, replace each piece with a mapping. That is genuinely the MAP_FIXED semantic. SEC_RESERVE answers (2). (3) is a series of small adjustments at existing call sites.

The new layer

pkg/noun/platform/windows/wloom.{c,h} — all Win32 behind a small C interface. events.c, manage.c and disk.c call into it and never see a HANDLE.

The live loom is at most two mappings inside one placeholder reservation:

[0, map_i)      copy-on-write view of image.bin, protected read-only
[map_i, len_i)  view of a SEC_RESERVE pagefile section, committed on touch

Two design decisions carry most of the weight.

The volatile half is a section view, not private memory. Moving the image boundary means unmapping and remapping that half, and Windows cannot resize a private region from below without discarding it — the heap and stack would go with it. Section contents outlive their views. The view always sits at section offset == loom offset, so addresses are stable across boundary moves. Growth is safe because the pages absorbed into the image are exactly the ones _ce_patch_apply just wrote to disk.

The image is mapped PAGE_WRITECOPY, then protected down to PAGE_READONLY. Stores therefore still fault into u3e_fault instead of being silently privatized. The view must be created write-copy; a read-only view cannot be upgraded later.

VirtualAlloc2 / MapViewOfFile3 / UnmapViewOfFileEx are resolved with GetProcAddress from kernelbase.dll rather than linked — mingw's import libraries do not reliably cover them, and their absence on pre-1803 Windows degrades to u3o_no_demand rather than failing to link. No build.zig link or _WIN32_WINNT changes were needed.

Sparseness

The loom section is created SEC_RESERVE, so its pages are reserved rather than committed, and commit charge tracks what the ship actually uses.

• u3_wnd_loom_commit commits a range; u3_wnd_loom_fault commits a page if and only if it was reserved.
• u3m_fault tries the latter first, ahead of everything else. It cannot wait for u3e_fault: u3m_water runs before it and dereferences the road, which lives in the loom and may itself be an untouched page.
• The vectored handler now routes read faults inside the loom to u3m_fault as well, since a reserved page faults on read too. This matches libsigsegv, which always delivered both. Stray reads outside the loom are left alone.
• Ranges the kernel writes into are committed up front — a handler cannot rescue a kernel-mode access.
• The guard page is committed before being protected, and u3e_yolo skips reserved regions rather than committing the whole loom.

Sparseness survives a save because _ce_patch_save_page already skips pages the allocator marks u3a_free_pg, so untouched memory is never read back.

u3m_init takes the placeholder path regardless of u3o_no_demand — sparseness is worth having on its own, so that flag now only chooses whether the image is mapped or blitted.

Migration

disk.c mapped the stale loom anonymously and then mapped the old image over it with MAP_FIXED. On Windows both halves fail: the anonymous mapping is a fully committed section of loom size, and a mapping cannot be replaced in place. wloom grew a small region array so a stale loom can coexist with the live one at its own base.

The stale loom is read, not mapped. The stale image and the migrated snapshot are the same file — a migration reads the old image.bin and writes the new one back over it — and Windows keeps a file's section attached to the handle that created it, so any mapping holds that file against every later resize of it. No arrangement of mappings is correct while source and destination are one file. Reading costs commit charge for the image where a mapping would not, but a migration is a one-time operation.

Three ordering bugs fell out, all latent on POSIX:

• _disk_migrate_loom released the stale loom after u3m_save. _disk_migrate_h had already been fixed this way in dbe52c39da; its sibling had not.
• _ce_loom_unmapf dropped the live image view but left its descriptor open — and Windows keeps the section attached to that descriptor, so it has to be cycled.
• u3m_init runs a second time after a migration, because the process goes on to boot the ship. u3m_stop never released the loom, and MAP_FIXED never cared.

Deliberately not supported

• --swap errors out on Windows rather than failing obscurely. The ephemeral file is remapped a page at a time, which the 64KB allocation granularity precludes against a 16KB loom page.
• Snapshot validation keeps forcing u3o_no_demand on Windows, since it reads the loom in the window where the image is unmapped.

[matthew-levan]

Boot and run tests between optimized 32-bit builds (-Doptimize=ReleaseFast) of ml/64-mingw-sparse @ abb5d17 vs. ml/64 @ dbe52c3

---

Post fresh-boot (no pier restart) with --urth-loom 29 --loom 31:

ml/64-mingw-sparse

ml/64

After restarting pier (|exit then urbit zod --urth-loom 29 --loom 31):

ml/64-mingw-sparse

ml/64

After restarting pier with demand paging OFF (|exit then urbit zod --urth-loom 29 --loom 31 --no-demand

ml/64-mingw-sparse

ml/64

After restarting the pier results table (mars)

Snapshot is ~786 MB. --urth-loom 29 --loom 31.

Metric	ml/64-mingw-sparse	ml/64	sparse --no-demand	ml/64 --no-demand
Working set	29,824 K	781,788 K	782,020 K	782,020 K
Peak working set	29,824 K	781,788 K	782,028 K	782,020 K
Memory (private working set)	15,232 K	2,256 K	2,268 K	2,292 K
Memory (shared working set)	14,592 K	779,532 K	779,752 K	779,728 K
Commit size	190,460 K	170,868 K	170,816 K	170,904 K
Page faults	8,322	196,314	196,382	196,349
I/O read bytes	34,288	785,941,999	785,942,000	785,942,000

26× less resident memory, ~23,000× less file I/O at boot, 24× fewer page faults.

ml/64 reads the entire snapshot into the loom at startup — I/O read bytes is almost exactly the size of image.bin. The sparse branch maps it instead, so it reads 34 KB and only faults in the ~29 MB the ship actually touches.

The three right-hand columns agree to within 0.03% on every metric, which is the control: demand paging accounts for the whole difference, and --no-demand on the sparse branch reproduces ml/64 exactly — so the fallback path (also taken on Windows older than 10 1803) is unchanged.

Commit size is higher on sparse by ~20 MB, as expected: the image is mapped PAGE_WRITECOPY, so pages the ship dirties become private copies, which this column counts. ml/64 keeps the loom in one pagefile section, where the cost sits in shared working set instead.

The urth process is unaffected in all four runs (~35 MB working set, ~24k page faults, ~6 KB read).

The fresh-boot pair is confounded: a fake zod saves during boot, and the two samples are three minutes apart against a snapshot timer, so some of that 488 vs 910 MB gap is likely one process having saved and remapped while the other hadn't. The restart test is the controlled one.

These numbers don't measure sparseness: every column is per-process, and SEC_RESERVE only shows in system commit charge.

Boot time

ml/64-mingw-sparse

C:\Users\Administrator\Desktop\vere>powershell -Command "Measure-Command { .\zig-out\x86_64-windows-gnu\urbit.exe -F nec --urth-loom 29 --loom 31 -x -t }"
disk: loaded epoch 0i0
loom: mapped 2048MB
boot: protected loom
live: logical boot
boot: installed 2179 jets
boot: parsing %brass pill
disk: loaded epoch 0i0
loom: mapped 2048MB
boot: protected loom
live: mapped: MB/785.940.480
boot: installed 2179 jets


Days              : 0
Hours             : 0
Minutes           : 2
Seconds           : 6
Milliseconds      : 910
Ticks             : 1269105048
TotalDays         : 0.00146887158333333
TotalHours        : 0.035252918
TotalMinutes      : 2.11517508
TotalSeconds      : 126.9105048
TotalMilliseconds : 126910.5048

ml/64

C:\Users\Administrator\Desktop\vere-ref>powershell -Command "Measure-Command { .\zig-out\x86_64-windows-gnu\urbit.exe -F nec --urth-loom 29 --loom 31 -x -t }"
disk: loaded epoch 0i0
loom: mapped 2048MB
boot: protected loom
live: logical boot
boot: installed 2179 jets
boot: parsing %brass pill
disk: loaded epoch 0i0
loom: mapped 2048MB
boot: protected loom
live: loaded: MB/785.940.480
boot: installed 2179 jets


Days              : 0
Hours             : 0
Minutes           : 2
Seconds           : 8
Milliseconds      : 368
Ticks             : 1283686536
TotalDays         : 0.00148574830555556
TotalHours        : 0.0356579593333333
TotalMinutes      : 2.13947756
TotalSeconds      : 128.3686536
TotalMilliseconds : 128368.6536

[matthew-levan]

The Windows loom: a lifecycle walkthrough

How vere initializes, runs, and releases the snapshot with demand paging enabled on Windows. Written for someone who knows events.c and POSIX memory management, and not much Win32.

Paths are relative to the repository root.

0. Five Windows concepts

POSIX collapses these into mmap; Windows splits them, and nearly every design decision in this subsystem falls out of the split.

Concept	What it is	POSIX analogue
Section object	A kernel object representing backing store, created by CreateFileMapping. Backed by a file, or by the pagefile if you pass INVALID_HANDLE_VALUE.	the fd argument to mmap (or MAP_ANON)
View	An actual mapping of a section into your address space, via MapViewOfFile*.	the return value of mmap
Reserve vs commit	MEM_RESERVE takes address space only. MEM_COMMIT promises backing store, charged against RAM + pagefile.	no equivalent — Linux overcommits, so every mapping is effectively "reserved"
Placeholder	MEM_RESERVE_PLACEHOLDER — a reservation you are permitted to later split and replace with a view. Windows 10 1803+.	MAP_FIXED
VEH	Vectored exception handler — a callback for access violations.	SIGSEGV handler

The one that trips people up: a section can itself be reserved rather than committed (SEC_RESERVE). That is what makes the loom sparse.

Minimum Windows

VirtualAlloc2 and MapViewOfFile3 arrived in Windows 10 1803, and are exported from the api-ms-win-core-memory-l1-1-6 API set rather than from kernel32. Linking them (see the linkSystemLibrary calls in build.zig and pkg/noun/build.zig) therefore sets vere's floor at that release: Windows resolves imports eagerly at load, so an older system refuses to start the binary at all rather than degrading. That floor is deliberate.

1. Boot, part one: reserving the loom

u3m_init → u3_wnd_loom_init → _wnd_reserve.

_wnd_reserve makes three Win32 calls in order:

1. CreateFileMappingW(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE | SEC_RESERVE, …) — creates a pagefile-backed section the size of the whole loom. This is the loom's anonymous memory. SEC_RESERVE is the crucial flag: its pages are reserved, not committed, so creating it costs no commit charge. Without it, Windows charges the entire loom against RAM plus paging file immediately — that was the original ERROR_COMMITMENT_LIMIT failure on a small box.

2. VirtualAlloc2(NULL, bas_v, len_i, MEM_RESERVE | MEM_RESERVE_PLACEHOLDER, PAGE_NOACCESS, …) — reserves the loom's address range as a placeholder. This is the piece that makes everything else possible: a plain reservation cannot be replaced by a mapping, but a placeholder can.

3. MapViewOfFile3(sec_h, NULL, bas_v, 0, len_i, MEM_REPLACE_PLACEHOLDER, PAGE_READWRITE, …) — maps the section over the entire placeholder.

The function writes into the slot only on success (wloom.c:391), so a failed reservation leaves it free rather than half-claimed.

After this the loom is one view of a reserved section, with nothing committed. Functionally this is mmap(MAP_ANON|MAP_PRIVATE) on Linux-with-overcommit.

2. Boot, part two: mapping the image

u3e_live opens image.bin, applies any leftover patch, calls u3e_foul() to mark every page dirty, then reaches the demand-paging branch and calls the Windows _ce_loom_mapf.

That function is short and mostly arithmetic:

gan_w = granularity_bytes >> (u3a_page + u3a_word_bytes_shift);  // 64KB/16KB = 4
map_w = pgs_w & ~(gan_w - 1);   // granule-floored page count
tal_w = pgs_w - map_w;          // ragged tail, 0..3 pages

Placeholders can only be split at the 64KB allocation granularity, but a loom page is 16KB. So the image is mapped up to the last whole 64KB granule, and the leftover ≤3 pages are pread into memory by _ce_loom_blit_pages — which commits them first, because a kernel-mode write (ReadFile into your buffer) cannot be rescued by a fault handler.

The real work is _wnd_remap, the heart of the subsystem:

1. _wnd_image — CreateFileMappingW(fil_h, NULL, PAGE_WRITECOPY, …), a copy-on-write section over image.bin. Deliberately done before any teardown, so a failure here is not destructive.

2. _wnd_hold — collapse whatever is mapped back into one placeholder. UnmapViewOfFileEx(bas, MEM_PRESERVE_PLACEHOLDER) unmaps the view but keeps the address range reserved as a placeholder — that flag is the difference between "give the address space back" and "hold it for me". If an image view already existed, the high view is unmapped too and VirtualFree(…, MEM_RELEASE | MEM_COALESCE_PLACEHOLDERS) merges the two placeholders back into one.

3. VirtualFree(bas, byt_i, MEM_RELEASE | MEM_PRESERVE_PLACEHOLDER) — splits the placeholder at the image boundary. Same call, different flag, opposite meaning from step 2's coalesce.

4. MapViewOfFile3(img_h, …, MEM_REPLACE_PLACEHOLDER, PAGE_WRITECOPY, …) — image into the low placeholder.

5. VirtualProtect(bas, byt_i, PAGE_READONLY, &old_u) — immediately downgrade to read-only. This two-step is not optional. The view must be created write-copy so it is permitted to be dirtied privately; but it must run read-only so stores trap into u3e_fault for dirty tracking. A view created PAGE_READONLY can never be upgraded later.

6. MapViewOfFile3(sec_h, …, bas + byt_i, offset byt_i, len - byt_i, …) — the pagefile section over the remainder.

Two things worth internalising about step 6:

• The offset equals the loom offset. Loom address X always maps to section offset X. That is what keeps addresses stable when the boundary moves.
• The volatile half is a section view, not private memory. Moving the boundary means unmapping and remapping this half, and Windows cannot resize a private region from below without discarding it — the heap and stack would go with it. Section contents outlive their views.

3. Running: three kinds of fault

Everything enters through _windows_exception_filter. ExceptionInformation[0] is 0 for a read, 1 for a write; [1] is the faulting address. Returning EXCEPTION_CONTINUE_EXECUTION re-runs the faulting instruction, which is how "fix it and retry" works.

It calls u3m_fault, which handles three cases in a deliberate order.

(a) First touch of a reserved page — the sparse case. Resolved first, at manage.c:2331, via u3_wnd_loom_fault: VirtualQuery the address, and if State == MEM_RESERVE, VirtualAlloc(…, MEM_COMMIT, PAGE_READWRITE).

The ordering is load-bearing for two reasons. u3m_water runs right after and dereferences u3R — which lives in the loom and may itself be untouched, so a first touch there would fault inside the fault handler. And the loom bounds check comes after too, because a migration holds a stale loom outside the live loom's range.

(b) Write to a clean image page — the demand-paging case. Falls through to u3e_fault, which sets the dirty bit and calls _ce_flaw_mprotect. That is plain mprotect(PROT_READ|PROT_WRITE) — but the shim in compat.c VirtualQuerys the range and substitutes PAGE_WRITECOPY when AllocationProtect is write-copy, because VirtualProtect rejects PAGE_READWRITE on a CoW view. It keys off AllocationProtect, not Protect, since an already-copied page reports PAGE_READWRITE while its view is still write-copy.

(c) The guard page — unchanged from POSIX, except that gar_w == 0 now means unposted rather than page zero (events.c:250).

4. Saving

u3e_save, in order:

1. _ce_patch_compose — walk the dirty bitmap, hash and write dirty pages to memory.bin / control.bin. Allocator-free pages are skipped, which is why sparseness survives a save.
2. _ce_patch_sync, _ce_patch_verify.
3. _ce_loom_unmapf — the new hook, a no-op on POSIX. Two jobs: u3_wnd_loom_unmapf drops the image view (_wnd_remap with byt_i = 0, so the pagefile section covers the whole loom again), and it closes and reopens u3P.img_u.fid_i. Windows keeps a file's section attached to the handle that created it, so unmapping alone does not release the file — the descriptor has to be cycled.
4. _ce_patch_apply → _ce_image_resize → u3_wnd_truncate (SetFilePointerEx + SetEndOfFile, used instead of ftruncate so failures report a real Win32 error), then pwrites the patch pages into image.bin.
5. _ce_loom_mapf again — remap at the new boundary.

Step 3 exists entirely because Windows refuses to truncate a mapped file. On POSIX steps 3–5 are just mmap(MAP_FIXED) doing the right thing.

Step 5 is the boundary-move case, and the one still untested by wloom_probe.c.

5. Release

u3e_stop closes the fd and frees the bitmap; the loom itself goes away at process exit. If u3m_init runs a second time in the same process — which happens after a migration — u3_wnd_loom_init calls _wnd_release first, making the reservation idempotent the way MAP_FIXED always was.

_wnd_release unmaps both views and releases each half separately, since a split reservation is two allocations and VirtualFree(bas, 0, MEM_RELEASE) would only free the first.

Tracing it live

The two highest-value breakpoints:

• _wnd_remap — every mapping change goes through it.
• u3_wnd_loom_fault — every first touch.

The two places where Windows semantics are genuinely unlike anything in POSIX are the placeholder split/coalesce dance in _wnd_remap steps 2–3, and the copy-on-write protection two-step in steps 4–5.

[matthew-levan]

Now with u3_wnd_loom_toss, we "toss" unused memory back to the Windows kernel via DiscardVirtualMemory so it can reclaim it from the process's working set. You can see here in this screenshot that, after fresh booting a fakezod (and without restarting it) and idling in dojo for a few moments, the mars process only needs a fraction (~20MB) of the memory which was used for booting (~2GB).