portelli/Grid
1
0
Fork 0
mirror of https://github.com/paboyle/Grid.git synced 2026-08-29 13:59:36 +01:00
Code Releases Activity

Debug ulimit as core files driving me crazy

Browse Source
This commit is contained in:
Peter Boyle
2026-08-28 11:20:43 -04:00
parent 9b4c388585
commit 63c2cdb712
4 changed files with 66 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/gpu-runtime-correctness.md
+30
View File
@@ -99,3 +99,33 @@ Grid uses `computeStream` (CUDA/HIP) or `theGridAccelerator` (SYCL) consistently
3. If results are non-deterministic across runs, insert a second barrier and observe whether reproducibility improves.
4. For correctness-critical operations (reductions that will be compared against reference values), add the double-run checksum test from `correctness-verification.md`.
5. If the process hangs at 100% CPU in a runtime library function, this is a driver/runtime bug — there is no application-level fix beyond scheduling a node reboot.
## Making an asynchronous GPU fault attributable
A "Memory access fault by GPU node-N ... Reason: Unknown" is raised by the
runtime, asynchronously, possibly long after the offending kernel or copy
was *queued*. `--debug-signals` sees nothing useful: the host is elsewhere
by then. Before exhaustive logging or bisection, make the fault land on the
call that caused it:
| runtime | serialise launches/copies | name kernels as they launch |
|---|---|---|
| HIP/ROCm | `AMD_SERIALIZE_KERNEL=3 AMD_SERIALIZE_COPY=3` (every launch and copy synchronous) | `AMD_LOG_LEVEL=3` (4 is very verbose) |
| CUDA | `CUDA_LAUNCH_BLOCKING=1` | `compute-sanitizer --tool memcheck ./bin ...` (names the kernel and the bad address) |
| SYCL/L0 | `SYCL_PI_TRACE=2` / `ZE_DEBUG=1` | (Level Zero validation layer: `ZE_ENABLE_VALIDATION_LAYER=1`) |
5-10x slower, so run on the smallest reproducer (one node, small lattice).
The last kernel name printed before the fault is the culprit; combined with
Grid's `--log ...,Memory` (every MemoryManager transfer with size, direction,
device and host pointers, in program order) it separates "stale pointer after
eviction" from "out-of-range index" in one run. First used 2026-08-28 for the
NRHS>=6 fault in the 3-level example (`systems/Frontier/nrhs_fault.job`).
**GPU core dumps (ROCm):** on a fault the runtime writes `gpucore.<pid>` (a
GCD's memory, ~22 GB on MI250X) into the process's working directory; it obeys
`ulimit -c` exactly as the kernel core does, so `ulimit -c 0` in the batch
script suppresses it (Slurm propagates rlimits to `srun` tasks by default) and
`ulimit -c unlimited` + `HSA_ENABLE_DEBUG=1` produces one ROCgdb can load.
Deleted-while-open dumps on NFS persist as hidden `.nfs*` files against quota
until the holder exits. Run fault hunts from a Lustre directory.
Source: ROCgdb documentation, "AMD GPU" chapter, core-dump section.