Deadlock when GC traverses a Custom value borrowed across a blocking FFI call

[yamamel]

Summary

A blocking FFI call on a SteelVal::Custom can deadlock the garbage collector and, in an embedded UI, freeze the whole application.

Reproduction (Helix Steel integration)

This is reproducible with the current Helix Steel integration and the helix-file-watcher cog.

1. Enable both the forest dynamic component and file watcher in init.scm:

   (require "forest/forest.scm")
   (require "helix-file-watcher/file-watcher.scm")
   (forest-configure! 'left #:ignore (list ".git" "target" "__pycache__"))
   (forest-set-style! 'mini)
   (spawn-watcher)
   (keymap (global) (normal (space (o ":forest-open"))))

2. Start hx with at least one file open, in a directory with several entries. Do not modify a watched file; the watcher should remain blocked in its event receive.

3. Press Space o to open forest and use j/k (or h/l) repeatedly to move the selection. On this configuration the UI freezes after a few moves once allocation triggers GC.

4. From another terminal while it is frozen, capture the process:

   sample "$(pgrep -n hx)" 10 -fullPaths -file /tmp/hx-hang.txt

Restarting Helix is required after the freeze.

Full sample

The complete 107 KB Apple sample output is available here: https://gist.github.com/yamamel/1d45424c77d2d21b24c56cbed0fdc5f1.

Observed integration case

Helix runs a Steel file-watcher cog on a native Steel thread. Its event handle is effectively EventHandle(Mutex<Receiver<Event>>), and its recv(&self) FFI method locks that mutex and blocks in Receiver::recv().

While that call is blocked, navigating a Steel dynamic component (in this case forest.hx) eventually allocates enough to trigger GC. Helix then freezes completely.

Why it deadlocks

as_ffi_argument currently converts a SteelVal::Custom by acquiring c.try_write() and retaining that outer write guard in FFIArg::CustomRef for the duration of the FFI call, including calls whose Rust receiver is only &self:

steel/crates/steel-core/src/steel_vm/ffi.rs

Lines 1809 to 1835 in dec633b

	SteelVal::ByteVector(b) => Ok(FFIArg::ByteVector(b.vec.read().iter().copied().collect())),
	// We can really only look at values that were made from the FFI boundary.
	SteelVal::Custom(c) => {
	// let mut guard = if let Ok(guard) = RefCell::try_borrow_mut(c) {
	let mut guard = if let Ok(guard) = c.try_write() {
	guard
	} else {
	stop!(Generic => "value cannot be borrowed mutably twice over the ffi boundary: {:?}", value)
	};
	if let Some(c) = as_underlying_type_mut::<OpaqueFFIValueReturn>(guard.as_mut()) {
	// SAFETY:
	// This should only be called internally, and the scope of the lifetime should be limited
	// to macro generated code.
	// TODO: I think it is possible that if the same value is used multiple times,
	// we could be borrowing this value more than once. We need a stickier way
	// to borrow since the RefCell cannot be passed along?
	unsafe {
	Ok(FFIArg::CustomRef(CustomRef {
	custom: RMut::from_raw(&mut c.inner as *mut _),
	guard,
	}))
	}
	} else if let Some(c) = as_underlying_type_mut::<FFIVector>(guard.as_mut()) {
	unsafe {
	let mut_ptr = &mut c.vec as *mut RVec<FFIValue>;

GC later traverses custom values by taking a shared lock before calling visit_children:

steel/crates/steel-core/src/values/closed.rs

Lines 323 to 330 in dec633b

	fn visit_custom_type(&mut self, custom_type: GcMut<Box<dyn CustomType>>) -> Self::Output {
	let mut queue = MarkAndSweepContext {
	queue: &mut self.queue,
	stats: MarkAndSweepStats::default(),
	};
	custom_type.read().visit_children(&mut queue);
	}

Therefore:

1. the watcher thread holds the outer custom-value write lock while waiting indefinitely for a file event;
2. the UI thread triggers GC and waits for the corresponding read lock;
3. neither can make progress.

An Apple sample captured during the freeze shows the UI thread in every sample at Synchronizer::enumerate_stacks -> MarkAndSweepContext::visit_custom_type -> parking_lot::RawRwLock::lock_shared_slow. The watcher thread is in every sample inside the FFI call, waiting at EventHandle::recv -> Receiver::recv.

Expected behavior

A blocking native method with an immutable receiver should not prevent GC from traversing the custom value and should not freeze the embedding UI.

Possible direction

The FFI conversion likely needs to preserve mutability in its locking representation: use a shared outer guard for &T receivers and retain the exclusive guard for &mut T receivers. Merely skipping locked custom values during GC would be unsafe because custom types can contain reachable Steel child values.

The file watcher can also avoid holding a call across an indefinite wait (for example, poll/timeout and yield), but the &self-to-exclusive-guard behavior appears to be the underlying Steel-side issue.

Environment

• Steel: dec633b908afeafeaf62bab457a92e2bf873745a
• Helix/Steel integration on macOS arm64

[mattwparas]

Tricky one! Latest commit of the file watcher switches to polling to avoid this while I figure out how to fix this in steel

[yamamel]

Tricky one! Latest commit of the file watcher switches to polling to avoid this while I figure out how to fix this in steel

Thanks a lot! I am using the latest file watcher and helix does not get stuck again.