GH-126910: Add gdb support for unwinding JIT frames#146071
GH-126910: Add gdb support for unwinding JIT frames#146071pablogsal merged 39 commits intopython:mainfrom
Conversation
|
🤖 New build scheduled with the buildbot fleet by @diegorusso for commit ac018d6 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146071%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
|
I have some questions about the EH frame generation and how it applies to the different code regions. Looking at
Both end up calling The EH frame in I understand how this is correct for _Py_CODEUNIT *
_JIT_ENTRY(...) {
jit_func_preserve_none jitted = (jit_func_preserve_none)exec->jit_code;
return jitted(exec, frame, stack_pointer, tstate, ...);
}The compiler will emit exactly the prologue/epilogue the EH frame describes. But I don't understand how the same EH frame is correct for The FDE covers the full The test ( Am I missing something about how the stencils interact with the stack, or is the EH frame intentionally approximate for the executor region? |
| struct jit_code_entry *first_entry; | ||
| }; | ||
|
|
||
| static volatile struct jit_descriptor __jit_debug_descriptor = { |
There was a problem hiding this comment.
Should these be non-static? The GDB JIT interface spec says GDB locates __jit_debug_descriptor and __jit_debug_register_code by name in the symbol table. With static linkage they would be invisible in .dynsym on stripped builds and when CPython is loaded as a shared library via dlopen. Am I missing something, or would this silently break in release/packaged builds where .symtab is stripped?
Maybe also worth adding __attribute__((used)) to prevent the linker from eliding them?
There was a problem hiding this comment.
Yes, you are right. Instead of removing the static I've exported with the macro Py_EXPORTED_SYMBOL
| id(42) | ||
| return | ||
|
|
||
| warming_up = True |
There was a problem hiding this comment.
Could this loop hang? When warming_up=True, the call passes warming_up_caller=True which returns immediately at line 8, so the recursive body never actually executes. If the JIT does not activate via some other path, would this not spin forever until the timeout kills it? Should there be a max iteration count as a safety net?
Also, line 16 uses bitwise & instead of and. Was that intentional? It means is_active() is always evaluated even when is_enabled() is False.
There was a problem hiding this comment.
I've simplified the test, the loop is not more controlled and deterministic.
| return; | ||
| } | ||
| _PyJitUnwind_GdbRegisterCode( | ||
| code_addr, (unsigned int)code_size, entry, filename); |
There was a problem hiding this comment.
code_size comes in as size_t but gets cast to unsigned int here. I know JIT regions will not be 4GB, but should the API just take size_t throughout for consistency?
There was a problem hiding this comment.
This is now done.
What this change synthesises for jit_executor is one unwind description for the executor as a whole, not compiler-emitted per-stencil CFI. Because the stencils are musttail-chained, the jumps between stencils do not add extra native call frames. The unwind job here is just to recover the caller of the executor frame. We don't want to describe each stencil as its own frame. When GDB stops at a PC inside
On AArch64, for most of the covered executor range, the synthetic CFI says:
Good catch for the testing gap. I’ve now added a new test that breaks inside the jit executor. It sill breaks at the |
|
@diegorusso @pablogsal I think I may have come up with a solution that works. EDIT: I think I gdb doesn't only use Background info (skip if not interested):
The current issue:
The solution:
This should work with TLDR: frame pointers = eh_frame is simple. |
@diegorusso I have to say that I am tremendously confused here. If GDB or So the real question is not “does the FDE cover the address range?” and it is not “do the stencils form one logical frame?”. The real question is: does the CFI row that applies at that PC actually describe the machine state there? That is the part I do not think has been explained. I agree with the narrow And that is exactly where I think the argument goes off the rails.
A concrete x86_64 example of why this seems wrong to me: with the same sort of flags used for executor stencils, a or, if it needs temporary stack space / spills, something more like In the first case there is no For Also, I rebuilt the branch locally and tried the exact “finish to So this is not just a theoretical concern for me. I still do not understand why the model being described here is supposed to work.I am of course not objecting to the goal. I am saying I still do not see the correctness argument. If the claim is that this is actually a correct unwind description for |
Thanks, thanks for the comment. I regenerated the x86_64 and AArch64 stencils after the recent frame-pointer changes. What we have today is that shim gets a real frame-pointer prologue, but the executor stencils still are not uniformly rbp/x29-framed, so I don’t think the current generated code is enough to justify a single executor-wide CFA = rbp + 16 / x29 + const rule for arbitrary PCs in the blob. The current implementation is still one synthetic executor-wide FDE. The unwinder uses the current PC to select that FDE and apply its CFI to recover the caller frame. That works where the actual machine state matches the synthetic rule at the stop PC, but it is still approximate executor-wide unwind metadata, not exact per-stencil CFI. Separately, once this PR lands, wiring up libgcc-backed backtrace should be fairly easy. We already synthesise .eh_frame; the remaining work is to call the appropriate |
Ok, I think now I understand. After re-checking the generated stencils I agree the current explanation was too bold. musttail only establishes the narrow point that the stencil-to-stencil transitions do not accumulate one native call frame per stencil and it does not by itself establish the stronger property needed for unwinding: that for an arbitrary PC inside jit_executor, the CFA and saved return state always have a shape described by one executor-wide FDE. That stronger property is the missing invariant here. After looking again at the regenerated x86_64 and AArch64 stencils, I don't think we have that invariant today:
I cannot justify the current synthetic executor-wide FDE as being correct for arbitrary PCs in the executor blob. The new test I added is still useful, but it proves something narrower: that the synthetic FDE works for the exercised in-executor stop. It does not prove that the same CFI is exact for every interior PC in the region (like you did in your example) I think the real options are:
The current implementation does not yet have the invariant needed to justify one executor-wide FDE for jit_executor but at the same time I don't really like the suggestions above. Let me think about it |
The current generation reserve the rbp. So all current stencils assume an rbp. Do you think it would fix it if we emitted our own prologue for the very first JIT executor uop ie (
Unfortunately, it seems you're right here. I dug around libgcc a little more and that's the only interface I see that intercepts |
Not all of them. See I'm not entirely sure your statement is true. |
Huh that's surprising! On x86_64, the current |
Oh sorry I'm wrong, not to it to get things like that. |
Accept either JIT backtrace shape in test_jit.py: - py::jit:executor -> _PyEval_* - py::jit:executor -> _PyJIT_Entry -> _PyEval_* This avoids baking in architecture-specific unwind details while still checking that GDB gets out of the JIT region and back into the eval loop.
Documentation build overview
100 files changed ·
|
markshannon
left a comment
There was a problem hiding this comment.
I've just tried this out and it works nicely (on linux AArch64). Here's an example:
Hitting ctrl-C while running this under gdb:
def count(seq):
t = 0
for i in seq:
t += math.sqrt(1.0)Gives me this backtrace:
#0 PyFloat_AsDouble (op=0xfffff7840770) at Objects/floatobject.c:253
#1 0x0000fffff7779074 in math_1 (err_msg=0xfffff777bf38 "expected a nonnegative input, got %s", can_overflow=0, func=<optimized out>, arg=<optimized out>) at ./Modules/mathmodule.c:792
#2 math_sqrt (self=<optimized out>, args=<optimized out>) at ./Modules/mathmodule.c:1161
#3 0x0000fffff7fe72f0 in py::jit:executor ()
#4 0x0000aaaaaab57960 in _PyEval_EvalFrameDefault (tstate=0xaaaaab31b3c0, frame=0xfffffffffffffffd, throwflag=-1424409936, throwflag@entry=0) at Python/generated_cases.c.h:5941
#5 0x0000aaaaaad3a96c in _PyEval_EvalFrame (throwflag=0, frame=0xfffff7fea020, tstate=0xaaaaab1c7de8 <_PyRuntime+344816>) at ./Include/internal/pycore_ceval.h:122
...
and I can use the up, down, step and continue commands as I would expect.
Stepping into the jitted code even works (sort of):
Single stepping until exit from function py::jit:executor,
which has no line number information.
Generate per-target JIT unwind constants from the compiled shim EH frame, emit dispatcher headers for the unwind info, and update the GDB JIT unwind tests.
| #else | ||
| # error "Unsupported target architecture" | ||
| #endif | ||
| DWRF_U8(DWRF_CFA_def_cfa); |
pablogsal
left a comment
There was a problem hiding this comment.
Thank you very much, Diego. This PR has been quite a Sisyphean effort, and the result looks very solid.
The DWARF writing is very elegant, in particular deriving the unwind info from the shim CFI instead of hard-coding another copy of the rules. Excellent job ❤️ 👌
I found and fixed two small issues before landing:
- Reset the JIT GDB mutex earlier after fork() so child-side executor cleanup cannot try to unregister JIT code while inheriting a locked mutex.
- Serialized perf jitdump initialization so concurrent first-use writers cannot observe a partially initialized jitdump state.
I’m landing this now.
Great work man! 🚀
The PR adds the support to GDB for unwinding JIT frames by emitting eh frames.
It reuses part of the existent infrastructure for the perf_jit from @pablogsal.
This is part of the overall plan laid out here: #126910 (comment)
The output in GDB looks like: