v0.77.0-rc.7: ISS reported despite 100% node agreement on state hash

[poulok]

v0.77.0-rc.7
BN = v0.40.0-rc1
Chicago's namespace chewie-solo-sdlt-n7/network-node1-0
Log: https://perf.analytics.eng.hashgraph.io/ephemeral/test_runs/_AdHoc7_186_16:13:40:08-14-2026/report/podlog_chewie-solo-sdlt-n7/network-node1_logs/swirlds.log
https://perf.analytics.eng.hashgraph.io/ephemeral/test_runs/_AdHoc7_186_16:13:40:08-14-2026/report/podlog_chewie-solo-sdlt-n7/network-node1_logs/hgcaa.log

2026-08-14 03:14:23.168 46       INFO  PLATFORM_STATUS  <platformForkJoinThread-6> StatusStateMachine: Platform spent 1.5 s in STARTING_UP. Now in REPLAYING_EVENTS {"oldStatus":"STARTING_UP","newStatus":"REPLAYING_EVENTS"} [com.swirlds.logging.legacy.payload.PlatformStatusPayload]
2026-08-14 03:14:23.171 47       INFO  STARTUP          <main> PcesCoordinator: replaying preconsensus event stream starting at 90184
2026-08-14 03:14:23.235 48       INFO  STARTUP          <<scheduler ConsensusEngine>> ConsensusImpl: Found init judge (CR:4 H:b4235a5f8406 BR:90209), num remaining: 6
2026-08-14 03:14:23.236 49       INFO  STARTUP          <<scheduler ConsensusEngine>> ConsensusImpl: Found init judge (CR:2 H:d11788875691 BR:90209), num remaining: 5
2026-08-14 03:14:23.236 50       INFO  STARTUP          <<scheduler ConsensusEngine>> ConsensusImpl: Found init judge (CR:1 H:3c6b9cbc30c1 BR:90209), num remaining: 4
2026-08-14 03:14:23.236 51       INFO  STARTUP          <<scheduler ConsensusEngine>> ConsensusImpl: Found init judge (CR:5 H:6a13c8ddf6d0 BR:90209), num remaining: 3
2026-08-14 03:14:23.236 52       INFO  STARTUP          <<scheduler ConsensusEngine>> ConsensusImpl: Found init judge (CR:0 H:8f40fd226ea4 BR:90210), num remaining: 2
2026-08-14 03:14:23.236 53       INFO  STARTUP          <<scheduler ConsensusEngine>> ConsensusImpl: Found init judge (CR:3 H:54b6c0dcea22 BR:90210), num remaining: 1
2026-08-14 03:14:23.236 54       INFO  STARTUP          <<scheduler ConsensusEngine>> ConsensusImpl: Found init judge (CR:6 H:3a7acd903451 BR:90210), num remaining: 0
2026-08-14 03:14:26.186 171      INFO  STARTUP          <<platform-core: MetricsThread #0>> LegacyCsvWriter: CsvWriter: Initializing statistics output in CSV format [ csvOutputFolder = 'data/stats', csvFileName = 'MainNetStats0.csv' ]
2026-08-14 03:14:34.135 500      FATAL EXCEPTION        <platformForkJoinThread-1> DefaultIssDetector: Invalid State Signature (ISS): this node has the wrong hash for round 95432.
Nodes holding 3500/3500 (100.0%) of total weight have reported a hash for round 95432.
Consensus mnemonic: acoustic-swamp-forward-shift
Consensus hash: f7f0c92732de91d2c71f064132091cf73b56e7883d1a06d490e7e51f9cb163cfb556c250a86d7e21a82187a0b619010c
- nodes 0, 1, 2, 3, 4, 5, 6
  partition weight: 3500 (100.0% of total weight)
  partition mnemonic: acoustic-swamp-forward-shift
  partition hash: f7f0c92732de91d2c71f064132091cf73b56e7883d1a06d490e7e51f9cb163cfb556c250a86d7e21a82187a0b619010c
 {"round":95432,"selfHash":"increase-focus-tunnel-lesson","consensusHash":"acoustic-swamp-forward-shift","catastrophic":false} [com.swirlds.logging.legacy.payload.IssPayload]
2026-08-14 03:14:34.137 501      INFO  PLATFORM_STATUS  <platformForkJoinThread-7> StatusStateMachine: Platform spent 11.0 s in REPLAYING_EVENTS. Now in CATASTROPHIC_FAILURE {"oldStatus":"REPLAYING_EVENTS","newStatus":"CATASTROPHIC_FAILURE"} [com.swirlds.logging.legacy.payload.PlatformStatusPayload]
2026-08-14 03:14:35.245 502      WARN  STARTUP          <platformForkJoinThread-3> HealthMonitorLogger: Task scheduler TransactionHandler has been unhealthy for 1.0 s. It currently has 152000/100000 unhandled tasks.
2026-08-14 03:16:44.133 503      INFO  STARTUP          <main> PcesReplayer: Replayed 82,711 preconsensus events with max birth round 98,373. These events contained 1,799,366 transactions. 5,221 rounds reached consensus spanning 7.7 minutes of consensus time. The latest round to reach consensus is round 95,432. Replay took 2.3 minutes.

[alex-kuzmin-hg]

@poulok All logs of all nodes: https://perf.analytics.eng.hashgraph.io/ephemeral/test_runs/_AdHoc7_186_16:13:40:08-14-2026/report/podlog_chewie-solo-sdlt-n7/

[poulok]

Timeline of events:

1. have all node run OK with Longevity tests, with small state of 100K acccounts, running for only 3 mins
2. while keep loading with Longevity
3. Round 95,432 is reached, state root hash is acoustic-swamp-forward-shift
4. nicely shutdown node 0 (with s6-svc -d)
5. wait 30 secs
6. start node0
7. node0 loads the last state on disk (round 90,211)
8. node0 replaces all PCES on disk and reaches consensus on round 95,432, but now it's root hash is increase-focus-tunnel-lesson, which conflicts with the state hash signatures created prior to the restart. The difference is in the first and last leaf path of VirtualMapMetadata:

-Root hash:                     a46b4ad1e8ac5c038ffaa05d1403e0a66957db49ae2daebfe807c88aa031b720649db62f30839d7d441928abe76089d9
+Root hash:                     f7f0c92732de91d2c71f064132091cf73b56e7883d1a06d490e7e51f9cb163cfb556c250a86d7e21a82187a0b619010c

-(root) VirtualMap    state    /    increase-focus-tunnel-lesson
+(root) VirtualMap    state    /    acoustic-swamp-forward-shift

-"VirtualMapMetadata":{"firstLeafPath":301223961,"lastLeafPath":602447922},
+"VirtualMapMetadata":{"firstLeafPath":300698526,"lastLeafPath":601397052},

[artemananiev]

This looks like a bug in MerkleDbDataSource. Here is what happens:

1. A state snapshot is written to disk
2. Then the node is shut down and restarted
3. At startup it loads the snapshot. A new MerkleDb data source instance is created
4. Here is the bug: all files from the snapshot dir are hard-linked to the new data source dir, which is in swirlds-tmp. This includes the metadata file. Once any data is flushed to the data source, it updates the metadata file, and the corresponding file in the snapshot dir is updated, too
5. Then for some reason the node is shut down and restarted from the same state snapshot
6. The snapshot dir is not touched, but the metadata file in it is updated. The restored MerkleDb instance has wrong metadata, which will lead to ISS in the future

The fix is straightforward. If a MerkleDb instance is restored (not created from scratch), it should delete the files it doesn't use: hash index, KV index, DB metadata. This is done in the data source dir in swirlds-tmp, the state snapshot dir will not be affected, only hard-links will be removed.