Skip to content

Recover a wedged W5500 link instead of sitting there unreachable#115

Open
tombueng wants to merge 1 commit into
masterfrom
fix/w5500-recovery
Open

Recover a wedged W5500 link instead of sitting there unreachable#115
tombueng wants to merge 1 commit into
masterfrom
fix/w5500-recovery

Conversation

@tombueng

Copy link
Copy Markdown
Owner

The problem

The W5500 (SPI Ethernet) sometimes comes up — or comes back after sustained RX, e.g. right after an OTA — reporting no link even though the cable is fine. It's the SPI RX-RSR erratum. A plain reboot doesn't clear it, so the box just sat there: DMX still clocking, but unreachable on the network, until someone power-cycled it or pulled the cable. (Hit exactly this on the bench today after an OTA reboot; recovered it with a cable-pull.)

Two gaps let it strand:

  • The runtime link watchdog only armed after the link had come up once (else if (wiredUp)), so a boot that never linked was ignored forever.
  • Nothing ever re-initialised the chip — it only waited for lwIP, which can't fix a wedged W5500.

The fix

ethReinit() tears the driver down (ETH.end), hard-resets the chip, and brings it back up (ETH.begin) on core 0 — the same core-affine path as the boot bring-up, so it can't reintroduce the DMX/RDM core contention from #64. A fresh begin re-negotiates the link, which is what actually clears the wedge (a bare RSTn pulse on its own doesn't — measured on earlier bench work). It runs:

  • once at boot if the link never came up (bounded, so an unplugged box isn't held off its WiFi/AP fallback), and
  • from the runtime watchdog, now covering the never-linked case too, with a 12s → 24s → 48s … 5 min backoff so an unplugged box doesn't thrash end/begin.

It only adds recovery attempts — no existing link-loss policy changes. /info.json exposes ethRecoverAttempts and ethRecoveries so a recovery is visible.

Validation (S3 + W5500 HIL rig)

Cut the link with the bench switch and held it 60s:

  • the watchdog fires the re-init (ethRecoverAttempts 0 → 2 with backoff, 0 → 4 without),
  • DMX keeps clocking straight through the core-0 re-init (analyzer seq advancing the whole time — the WT32-ETH01 DMX framing errors with Swisson tester #64 core-contention concern is handled),
  • the counters move, the backoff spaces the attempts,
  • and the box recovers when the link returns (ethRecoveries ticked on a re-init that coincided with the link coming back).

Happy path unaffected: 38 plain reboots + 6 OTA cycles today all came back clean with ethRecoverAttempts staying 0 — no false-firing.

Honest caveat: the RSR wedge itself is rare and load-triggered (0 reproductions in those 44 boots; 1 occurrence before that), so the re-init clearing that specific failure is reasoned, not reproduced — the mechanism is proven and the reasoning is that today's wedge cleared via a link event and ethReinit re-negotiates the link identically. It's now instrumented, so a real recovery in the field will show up in the counters. Since I couldn't reproduce the wedge, the behavioural proof is the rig link-cut (like signal-loss STOP and the DHCP-hostname cases, this needs the HIL switch, not Playwright); web-ui.spec.mjs pins the /info.json shape.

The W5500 can come up (or come back after sustained RX, e.g. right after an OTA)
reporting no link even though the cable is fine — the SPI RX-RSR erratum. A plain
reboot doesn't clear it, so the box just sat there: DMX still clocking, but
unreachable on the network, until someone power-cycled it or pulled the cable.
Two gaps let that happen: the runtime link watchdog only armed after the link had
come up once, so a boot that never linked was ignored forever; and nothing ever
re-initialised the chip, it only waited for lwIP.

Fix: ethReinit() tears the driver down (ETH.end), hard-resets the chip and brings
it back up (ETH.begin) on core 0 — the same core-affine path as the boot bring-up,
so it can't reintroduce the DMX/RDM core contention from issue #64. A fresh begin
re-negotiates the link, which is what actually clears the wedge (a bare RSTn pulse
on its own doesn't — measured earlier). It runs once at boot if the link never
came up, and from the runtime watchdog (now covering the never-linked case too),
with a 12s→24s→48s…5min backoff so an unplugged box doesn't thrash end/begin. It
only ADDS recovery attempts; no existing link-loss policy changes.

/info.json exposes ethRecoverAttempts and ethRecoveries so a recovery is visible.

Validated on the S3 + W5500 rig by cutting the link with the bench switch: the
watchdog fires the re-init, DMX keeps clocking straight through the core-0 work,
the counters move, the backoff spaces the attempts, and the box recovers when the
link returns. The RSR wedge itself is rare and load-triggered (0 in 38 reboots +
6 OTA cycles today, 1 occurrence before that), so the re-init clearing that
specific failure is reasoned, not reproduced — but it's now instrumented, so a
real recovery in the field shows up in the counters.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants