Skip to content

[Nexthop][fboss2-dev] Add fboss2-dev config/delete qos default-policy commands#1366

Open
vybhav-nexthop wants to merge 1 commit into
facebook:mainfrom
nexthop-ai:nos-7165-qos-default-policy
Open

[Nexthop][fboss2-dev] Add fboss2-dev config/delete qos default-policy commands#1366
vybhav-nexthop wants to merge 1 commit into
facebook:mainfrom
nexthop-ai:nos-7165-qos-default-policy

Conversation

@vybhav-nexthop

@vybhav-nexthop vybhav-nexthop commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Summary

Adds two CLI commands for the switch-wide default QoS policy (the policy applied to ports that don't have a per-port override):

Command What it does Commit level
fboss2-dev config qos default-policy <name> Sets sw.dataPlaneTrafficPolicy.defaultQosPolicy. Validates the name exists in qosPolicies. HITLESS
fboss2-dev delete qos default-policy Clears that field. Lives under delete qos, consistent with delete interface, delete protocol, etc. AGENT_COLDBOOT (see below)

Usage:

$ fboss2-dev config qos default-policy default
Successfully set default QoS policy to 'default'
$ fboss2-dev config session commit          # HITLESS

$ fboss2-dev delete qos default-policy
Successfully removed default QoS policy 'default'
$ fboss2-dev config session commit          # AGENT_COLDBOOT — restarts agents

This PR only adds CLI and tests. It does not fix the underlying hw-agent bug.


Known issue: hitless unset corrupts hw-agent state (pre-existing on main)

Found on the device while testing these commands.

Repro (deterministic)

config qos default-policy default + commit   → OK
delete qos default-policy         + commit   → OK, but hw agent is now silently broken
config qos default-policy default + commit   → hw_agent SIGABRT → forced cold boot (~50s outage)

Crash in the agent log:

Terminated due to: Failed to remove QoS map: none programmed default
    @ SaiQosMapManager::removeQosMap
    @ SaiSwitch::processDelta<...>
    @ SaiSwitch::stateChangedImplLocked
    @ OperDeltaSyncer::operSyncLoop

The same crash can happen on any later change to that policy in the map (edit maps, remove from qosPolicies) — not just re-setting it as default. The hitless delete creates the bug; later commits cause the crash.

Why unsetting the default is special

A QoS policy can live in one of two places in agent state:

  1. QosPolicyMap — named policies used by per-port overrides
  2. defaultDataPlaneQosPolicy — the switch-wide default

A policy is in exactly one place. When defaultQosPolicy is set in config, ThriftConfigApplier::updateQosPolicies() skips that policy when building the map (it lives in the default slot instead):

if (defaultDataPlaneQosPolicyName == *qosPolicy.name()) {
  continue;  // maintained in switch settings, not the map
}

Clearing defaultQosPolicy does not remove the policy from qosPolicies in config — it only clears one string. On the next commit, the applier stops skipping it, so one config change produces two agent deltas:

Before (default set) After (default cleared)
Config qosPolicies [{"name": "default"}] [{"name": "default"}] (unchanged)
Config defaultQosPolicy "default" (absent)
Agent QosPolicyMap {} {"default": ...}added
Agent default slot "default" (empty)removed

In short: picking a policy as default doesn't remove it from qosPolicies — CLI requires it stay there. Deleting the default just clears the defaultQosPolicy field. With that field now empty, the applier stops skipping the policy, so it gets added back into QosPolicyMap on the same commit that removes it as default.

What goes wrong in the hw agent

The hw agent tracks programmed QoS maps in SaiQosMapManager::handles_, whose key is the policy name only (one slot per name). It applies the two deltas in order:

processDelta(delta.getQosPoliciesDelta(), ...);       // (1) map add  → programs handle["default"]
processDefaultDataPlanePolicyDelta(delta, ...);       // (2) default remove → removeQosMap(..., true)

Step (2) should tear down the old default-slot programming. But removeQosMap ignores the isDefault flag and always erases by name:

void SaiQosMapManager::removeQosMap(..., bool /*isDefault*/) {
  handles_.erase(qosPolicyName);  // wipes the handle step (1) just created
}

The commit succeeds, but hw and sw agent don't have the same state: SwitchState says the policy is in the map, while handles_ has no entry for it.

isDefault is stored on each handle and used for default-policy lookup, but remove never checks it — so the flag doesn't protect this handoff. Unit tests that unset the default only clear the default slot from the defaultQosPolicy (one delta); they don't also add the policy to QosPolicyMap, so they never exercise this path.

Why delete uses AGENT_COLDBOOT

  • HITLESS keeps the hw agent running with the corrupted in-memory handles_ table — that's the failure mode.
  • COLDBOOT restarts agents and rebuilds handles_ from scratch. Config and SwitchState are already correct; only the in-memory table is wrong.

Set stays HITLESS — setting the default is safe on its own. It only crashes after a prior hitless unset left the agent in a bad state, which the delete command avoids by forcing COLDBOOT.


Test Plan

Unit tests, rerun fresh (14/14):

[==========] Running 14 tests from 2 test suites.
[----------] 10 tests from CmdConfigQosDefaultPolicyTestFixture (719 ms total)
[----------]  4 tests from CmdDeleteQosDefaultPolicyTestFixture (291 ms total)
[==========] 14 tests from 2 test suites ran. (1010 ms total)
[  PASSED  ] 14 tests.

Integration on NH-4010-F, rerun fresh (2/2):

[==========] Running 2 tests from 1 test suite.
[ RUN      ] ConfigQosDefaultPolicyTest.SetAndVerifyDefaultPolicy
[       OK ] ConfigQosDefaultPolicyTest.SetAndVerifyDefaultPolicy (21958 ms)
[ RUN      ] ConfigQosDefaultPolicyTest.DeleteAndVerifyDefaultPolicy
[       OK ] ConfigQosDefaultPolicyTest.DeleteAndVerifyDefaultPolicy (24742 ms)
[==========] 2 tests from 1 test suite ran. (46701 ms total)
[  PASSED  ] 2 tests.

Integration tests use COLDBOOT on delete, so they do not exercise the hitless-unset bug path. If tests pass but you see Channel got EOF or NRestarts incrementing on fboss_hw_agent@0 / fboss_sw_agent, the harness retried until agents recovered after a crash.

Review Findings

Pre-publication automated review flagged 4 potential missing null checks on
thrift field accessors (agentConfig.sw(), switchConfig.qosPolicies());
all verified false positives — both are unqualified thrift fields
(qosPolicies = [] default), so field_ref dereference is always valid.
No findings above threshold.

@vybhav-nexthop vybhav-nexthop requested review from a team as code owners July 7, 2026 15:03
@meta-cla meta-cla Bot added the CLA Signed label Jul 7, 2026
config qos default-policy <name> sets
sw.dataPlaneTrafficPolicy.defaultQosPolicy after validating the policy
name exists in qosPolicies. Commits at HITLESS (applied via
updatePort() in ApplyThriftConfig, no ChangeProhibited in SAI).

delete qos default-policy clears that field, under the delete subtree
consistent with delete interface / delete protocol. The delete commits
at AGENT_COLDBOOT rather than HITLESS to avoid a hw agent qos-map
handle desync on hitless unset (details in PR description).

Unit tests: 14 passing across set and delete fixtures.
Integration test: covers set and delete round-trip, restores prior
state.
@rabbit-nexthop rabbit-nexthop force-pushed the nos-7165-qos-default-policy branch from 92bbf16 to b5eefe4 Compare July 8, 2026 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant