Skip to content

Foreground SyncTriggers publish trigger metadata but never update the hash blob, breaking self-heal #11875

Description

@brettsam

Split out from #11873 (suggested fix #2).

Summary

FunctionsSyncManager.TrySyncTriggersAsync only reads and writes the SyncTriggers hash blob (synctriggers/{hostId}/last) when isBackgroundSync == true. Foreground syncs — inbound /admin/host/synctriggers requests — call SetTriggersAsync and publish the trigger payload to the platform, but they never check, update, or invalidate the hash blob. This leaves the hash blob out of sync with what was actually last published, which defeats the background-sync self-heal mechanism.

Details

In TrySyncTriggersAsync (FunctionsSyncManager.cs):

  • The hash is only consulted when isBackgroundSync is true:
    bool shouldSyncTriggers = true;
    string newHash = null;
    if (isBackgroundSync && hashBlobClient != null)
    {
        newHash = await CheckHashAsync(hashBlobClient, payload.Content);
        shouldSyncTriggers = newHash != null;
    }
    
    if (shouldSyncTriggers)
    {
        var (success, error) = await SetTriggersAsync(payload.Content);
        if (success && newHash != null)   // newHash is always null for foreground syncs
        {
            await UpdateHashAsync(hashBlobClient, newHash);
        }
        ...
    }
  • For a foreground sync, newHash stays null, so SetTriggersAsync publishes the payload but UpdateHashAsync is never called.

Why this is a bug

Because a foreground sync can publish a payload that differs from the hash currently stored in the blob, the blob no longer reflects the last-published trigger metadata. A subsequent background sync recomputes the current hash, finds it matches the (now-stale) blob, concludes "nothing changed," and suppresses the corrective re-publish — even though the live platform view is wrong.

This is the mechanism that wedges an app on the placeholder WarmUp payload in #11873:

  1. A healthy background sync writes the real-functions hash to the blob.
  2. A bad foreground WarmUp publish overwrites the live trigger payload but leaves the blob unchanged (still the real-functions hash).
  3. The host recovers, recomputes the real-functions hash, compares to the blob → they match → background sync suppresses the re-publish.

Result: the platform stays on stale/incorrect trigger metadata until an external forced sync (portal Refresh, ARM host/default/sync, or another deploy) happens to re-publish.

Proposed fix

Make foreground syncs keep the hash blob consistent with what they publish, so background self-heal can still detect drift. Two options:

  • Update the hash blob after a successful foreground SetTriggersAsync (compute and store the published payload's hash), or
  • Invalidate/delete the hash blob on foreground publish, forcing the next background sync to re-publish.

Updating is preferable to invalidating so we don't trigger an unnecessary extra publish, but either restores the self-heal guarantee that the blob always reflects the last-published payload.

Related

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions