RuntimeError: dictionary changed size during iteration during request handling

[Dongoing]

PyHSS 1.0.2.

Hit intermittently while handling a burst of requests:

RuntimeError: dictionary changed size during iteration

A dict is being mutated while it's iterated (not input-dependent — shows up under concurrent requests). Iterating over a copy (list(d.items())) or holding the relevant lock during iteration would fix it. I can try to narrow down which dict if that helps — it wasn't obvious from the traceback which structure it was.

Line numbers below are against current master. The cause is the shared peer table activePeers being iterated without a snapshot while it is mutated by peer connect/disconnect:

lib/diameter.py
806  for peerKey, peer in activePeers.items():
840  for peerKey, peer in activePeers.items():
884  for peerKey, peer in activePeers.items():

activePeers gains and loses entries as peers connect and drop. When a peer connects or disconnects while one of these loops is running, the dict changes size mid-iteration and raises RuntimeError: dictionary changed size during iteration. Peer churn (repeated reconnects) surfaces it.

The fix is to iterate over a snapshot (for peerKey, peer in list(activePeers.items()):) or to hold the peer lock across these loops.