Recover a wedged W5500 link instead of sitting there unreachable#115
Open
tombueng wants to merge 1 commit into
Open
Recover a wedged W5500 link instead of sitting there unreachable#115tombueng wants to merge 1 commit into
tombueng wants to merge 1 commit into
Conversation
The W5500 can come up (or come back after sustained RX, e.g. right after an OTA) reporting no link even though the cable is fine — the SPI RX-RSR erratum. A plain reboot doesn't clear it, so the box just sat there: DMX still clocking, but unreachable on the network, until someone power-cycled it or pulled the cable. Two gaps let that happen: the runtime link watchdog only armed after the link had come up once, so a boot that never linked was ignored forever; and nothing ever re-initialised the chip, it only waited for lwIP. Fix: ethReinit() tears the driver down (ETH.end), hard-resets the chip and brings it back up (ETH.begin) on core 0 — the same core-affine path as the boot bring-up, so it can't reintroduce the DMX/RDM core contention from issue #64. A fresh begin re-negotiates the link, which is what actually clears the wedge (a bare RSTn pulse on its own doesn't — measured earlier). It runs once at boot if the link never came up, and from the runtime watchdog (now covering the never-linked case too), with a 12s→24s→48s…5min backoff so an unplugged box doesn't thrash end/begin. It only ADDS recovery attempts; no existing link-loss policy changes. /info.json exposes ethRecoverAttempts and ethRecoveries so a recovery is visible. Validated on the S3 + W5500 rig by cutting the link with the bench switch: the watchdog fires the re-init, DMX keeps clocking straight through the core-0 work, the counters move, the backoff spaces the attempts, and the box recovers when the link returns. The RSR wedge itself is rare and load-triggered (0 in 38 reboots + 6 OTA cycles today, 1 occurrence before that), so the re-init clearing that specific failure is reasoned, not reproduced — but it's now instrumented, so a real recovery in the field shows up in the counters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The W5500 (SPI Ethernet) sometimes comes up — or comes back after sustained RX, e.g. right after an OTA — reporting no link even though the cable is fine. It's the SPI RX-RSR erratum. A plain reboot doesn't clear it, so the box just sat there: DMX still clocking, but unreachable on the network, until someone power-cycled it or pulled the cable. (Hit exactly this on the bench today after an OTA reboot; recovered it with a cable-pull.)
Two gaps let it strand:
else if (wiredUp)), so a boot that never linked was ignored forever.The fix
ethReinit()tears the driver down (ETH.end), hard-resets the chip, and brings it back up (ETH.begin) on core 0 — the same core-affine path as the boot bring-up, so it can't reintroduce the DMX/RDM core contention from #64. A freshbeginre-negotiates the link, which is what actually clears the wedge (a bare RSTn pulse on its own doesn't — measured on earlier bench work). It runs:end/begin.It only adds recovery attempts — no existing link-loss policy changes.
/info.jsonexposesethRecoverAttemptsandethRecoveriesso a recovery is visible.Validation (S3 + W5500 HIL rig)
Cut the link with the bench switch and held it 60s:
ethRecoverAttempts0 → 2 with backoff, 0 → 4 without),ethRecoveriesticked on a re-init that coincided with the link coming back).Happy path unaffected: 38 plain reboots + 6 OTA cycles today all came back clean with
ethRecoverAttemptsstaying 0 — no false-firing.Honest caveat: the RSR wedge itself is rare and load-triggered (0 reproductions in those 44 boots; 1 occurrence before that), so the re-init clearing that specific failure is reasoned, not reproduced — the mechanism is proven and the reasoning is that today's wedge cleared via a link event and
ethReinitre-negotiates the link identically. It's now instrumented, so a real recovery in the field will show up in the counters. Since I couldn't reproduce the wedge, the behavioural proof is the rig link-cut (like signal-loss STOP and the DHCP-hostname cases, this needs the HIL switch, not Playwright);web-ui.spec.mjspins the/info.jsonshape.