vgif exposure is inconsistent between hosts of oracle-vm-32cpu-128gb-x86-64, nested virtualization jobs fail on hosts without it

[myasnikovdaniil]

Hi! I want to ask you help in issue with runners provided to the cozystack project. We use oracle-vm-32cpu-128gb-x86-64 runner class.

Our use case

Cozystack is a platform that runs VMs and tenant kubernetes clusters on top of kubernetes, so our e2e is nested virtualization all the way down:

layer	what it is
L0	Oracle host
L1	runner VM (VM.Standard.E4.Flex)
L2	sandbox talos nodes, qemu/kvm VMs on the runner, together they form cozystack cluster
L3	tenant kubernetes worker VMs, created by kubevirt inside sandbox cluster

Suites that build L3 are kubernetes-latest and kubernetes-previous, these names appear in numbers below. This e2e is one of our main merge gates and runs for almost every PR.

What is wrong

Hosts of this runner class are not equal. Two runners compared directly are same shape (VM.Standard.E4.Flex), same EPYC 7J13, same kernel 6.17.0-1019-oracle, and differ in one CPU flag, vgif, and nothing else (106 vs 105 flag words in lscpu): 32638015838 has it, 32638895581 doesn't. We read the bit from CPUID directly. The class sometimes lands on other EPYC models too, but as far as we can tell processor model is not what splits the outcomes.

In our sampling 4 of 18 launches (~22%) landed on host with vgif.

Outcome over 2026-07-28 to 2026-08-24, unit is one test suite instance (kubernetes-latest or kubernetes-previous in one run, the two tenant kubernetes suites named above), host classified by kvm_amd: block in dmesg kept in run artifacts, runs where the block was not captured are excluded:

hosts	suite instances	pass	fail
vgif present	26	22	4
vgif absent	118	0	118

Zero passes in 118 instances without the flag, across two independent populations (nightly and pull request runs). On 2026-08-18 alone the split shows within one day: 17 runs without the flag failed both suites, 3 runs with it, two of them passed everything. So it is also not the OCPU halving from #544 (we asked about that separately in #601 to check it was intended), same-sized hosts pass and fail side by side on the same day, and this flag is the only difference we found that tracks the outcome.

How it shows in the numbers: the failing part is exactly where L2 works as hypervisor and L3 VMs must boot and join tenant cluster. On hosts with vgif this step completes in 439-549 s. On hosts without it the same step never completes and runs to its full deadline (measured windows 1710-1736 s), runner at 53-73% idle with ~0% steal the whole time, so L2 guest is stalled while L1 has plenty of free capacity. Three full e2e runs that drew vgif hosts on unmodified main gave 46/46, 46/46 and 45/46 suites passing, on a lane that was failing 60-90% of runs for the preceding month. We checked large amount of other variables and possible causes along the way, investigation log is in cozystack/cozystack#3513, deliberately not pulling it in here because it is too much data.

Before cozystack/cozystack#3945 the flag was not measured explicitly but derivable from dmesg kept in artifacts, since #3945 we record svm features: list in runner-identity.txt on every run.

What vgif is

On AMD, hypervisor clears and sets GIF (global interrupt flag) with clgi/stgi around every vmrun. When hypervisor itself is a guest, these instructions trap to the parent, unless vCPU has vgif (CPUID 0x8000000A EDX bit 16) which keeps GIF switching in hardware. Without vgif every world switch made by L2 hypervisor costs extra vmexits to L1, and with our nesting this tax lands on every L3 VM entry and exit. We did not measure trap amplification at runner level directly, we already invested large amount of engineer time into this and measuring exit rates properly would take another couple of days and many shared runner hours, but correlation above doesn't depend on the mechanism.

If you want to verify yourself

Artifacts of the runs are public and downloadable with any github account, retention lasts until 2026-11 for the oldest runs used here. Every number above is re-derivable from them:

# classify one run
gh run download <run-id> --name cozyreport -D /tmp/r
tar xzf /tmp/r/cozyreport.tgz -C /tmp/r ./cozyreport/sandbox-host/dmesg.txt
grep 'kvm_amd:' /tmp/r/cozyreport/sandbox-host/dmesg.txt

# its suite outcomes
gh run download <run-id> --name chainsaw-report -D /tmp/c
grep -E 'kubernetes-(latest|previous)' /tmp/c/chainsaw-report.xml

Example runs: vgif present and clean 32651494733 and 32653907632, vgif absent with expired L3 join 32588459358, 32595562465, 32625385831. Full data appendix with per-run tables we can attach if useful.

What we ask

I understand vgif is not configurable from this repo or at instance launch, shape spec (compute shapes) doesn't go down to SVM sub-features at all. But today the flag is present or absent on identical hosts at random, and we want this uncertainty resolved:

1. vgif is guaranteed for this class. Then absent flag is a host-side bug for Oracle to fix, and we can provide run IDs and timestamps of every bad draw.
2. Nobody guarantees vgif. Then we stop relying on it, which for us means refactoring e2e and moving these jobs from shared runners to our own infrastructure, rough estimate is 10-15% increase of our dev infra budget, calculations can be provided if needed.

Either answer works, we just need to know which one it is. Also if you have escalation channel to Oracle for host-level issues, we'd appreciate this being raised there, we have no contract with Oracle to do it ourselves.

If this repo is not the right place for such request, please route us to the correct one.

Why only now

It worked before. Our e2e has flakiness of its own, so we can't say why it broke in the last month, host-dependent failures were hiding in that noise. Failure rate got bad enough that we spent couple weeks of engineer time digging to this point.

[koksay]

Hi @myasnikovdaniil, first of all, thanks for the detailed analysis!

As we run many VMs simultaneously, we get capacity errors, especially for the larger shapes. Therefore, we added a mechanism to try different availability zones and regions (as configured here). Here is an example from one of the runners:

> k logs cncf-ubuntu-32-128-x86-2w5bm-runner-46sjw -n arc-systems | egrep "attempting|success|out of capacity" 
2026/08/24 02:58:27 main.go:183: attempting launch: region=us-chicago-1 shape=VM.Standard.E5.Flex
2026/08/24 02:58:27 main.go:188: out of capacity: region=us-chicago-1 shape=VM.Standard.E5.Flex: creating instance: Error returned by Compute Service. Http Status Code: 500. Error Code: InternalError. Message: Out of host capacity.
2026/08/24 02:58:27 main.go:183: attempting launch: region=us-chicago-1 shape=VM.Standard.E5.Flex
2026/08/24 02:58:28 main.go:188: out of capacity: region=us-chicago-1 shape=VM.Standard.E5.Flex: creating instance: Error returned by Compute Service. Http Status Code: 500. Error Code: InternalError.  Message: Out of host capacity.
2026/08/24 02:58:28 main.go:183: attempting launch: region=us-chicago-1 shape=VM.Standard.E5.Flex
2026/08/24 02:58:28 main.go:188: out of capacity: region=us-chicago-1 shape=VM.Standard.E5.Flex: creating instance: Error returned by Compute Service. Http Status Code: 500. Error Code: InternalError. Message: Out of host capacity.
2026/08/24 02:58:28 main.go:183: attempting launch: region=us-ashburn-1 shape=VM.Standard.E5.Flex
2026/08/24 02:58:29 main.go:188: out of capacity: region=us-ashburn-1 shape=VM.Standard.E5.Flex: creating instance: Error returned by Compute Service. Http Status Code: 500. Error Code: InternalError. Message: Out of host capacity.
2026/08/24 02:58:29 main.go:183: attempting launch: region=us-ashburn-1 shape=VM.Standard.E5.Flex
2026/08/24 02:58:30 main.go:188: out of capacity: region=us-ashburn-1 shape=VM.Standard.E5.Flex: creating instance: Error returned by Compute Service. Http Status Code: 500. Error Code: InternalError. Message: Out of host capacity.
2026/08/24 02:58:30 main.go:183: attempting launch: region=us-ashburn-1 shape=VM.Standard.E5.Flex
2026/08/24 02:58:31 main.go:188: out of capacity: region=us-ashburn-1 shape=VM.Standard.E5.Flex: creating instance: Error returned by Compute Service. Http Status Code: 500. Error Code: InternalError.  Message: Out of host capacity.
2026/08/24 02:58:31 main.go:183: attempting launch: region=us-chicago-1 shape=VM.Standard.E4.Flex
2026/08/24 02:58:32 main.go:188: out of capacity: region=us-chicago-1 shape=VM.Standard.E4.Flex: creating instance: Error returned by Compute Service. Http Status Code: 500. Error Code: InternalError.  Message: Out of host capacity.
2026/08/24 02:58:32 main.go:183: attempting launch: region=us-chicago-1 shape=VM.Standard.E4.Flex
2026/08/24 02:58:32 main.go:188: out of capacity: region=us-chicago-1 shape=VM.Standard.E4.Flex: creating instance: Error returned by Compute Service. Http Status Code: 500. Error Code: InternalError.  Message: Out of host capacity.
2026/08/24 02:58:32 main.go:183: attempting launch: region=us-chicago-1 shape=VM.Standard.E4.Flex
2026/08/24 02:58:33 main.go:188: out of capacity: region=us-chicago-1 shape=VM.Standard.E4.Flex: creating instance: Error returned by Compute Service. Http Status Code: 500. Error Code: InternalError. Message: Out of host capacity.
2026/08/24 02:58:33 main.go:183: attempting launch: region=us-ashburn-1 shape=VM.Standard.E4.Flex
2026/08/24 02:58:34 main.go:214: instance launched successfully: region=us-ashburn-1 shape=VM.Standard.E4.Flex

I created some instances and got one with vgif on Ashburn AD-3, but not on AD-1 or AD-2. I've changed the ordering so you have a higher chance of getting it (hopefully).

My educated guess: Oracle can't/won't change this behavior; I assume that nobody will guarantee it. I will still create a ticket to them and get back to you with an official answer.

[koksay]

Or maybe it will not help at all, my test run also did not have vgif this time. I will have more info after the OCI support issue.

[myasnikovdaniil]

Thank you, we'll be waiting for updates from you, meanwhile I'll try to remove one level of virtualization from our tests.