ninja -C build test intermittently dies on a fatal signal — SIGSEGV or SIGILL — rather than
failing a test. It happens on main and on the mmtk branch with no PR involved, at two distinct
points in the suite, and it is currently the most common CI failure mode: across the last 30
failed test.yml runs, Run regression tests was the failing step 30 times (ANSI 13, koga 18
combined, Build 7).
I could not reproduce it locally in 30 attempts (details at the end), so this is a report rather
than a patch. The occurrence data below is verified job by job.
How to recognise it
| exit |
meaning |
tell |
code=1 |
ordinary test failure |
prints a Successes: NNNN summary |
code=132 |
SIGILL (128+4) |
no summary — the process died |
code=139 |
SIGSEGV (128+11) |
no summary |
A run that fails with no Successes: line has crashed, not failed. That distinction matters,
because these currently get retried as flaky infrastructure.
Confirmed occurrences
| run |
branch |
cell |
signal |
crash site |
| 31214144149 |
main |
clasp/macos/native/yes/no |
132 |
compiling snapshot.lisp |
| 30921569262 |
mmtk |
clasp/macos/native/yes/no |
139 |
after FINALIZERS-GENERAL-REMOVE |
| 30921569262 |
mmtk |
cando/macos/native/yes/no |
139 |
after FINALIZERS-GENERAL-REMOVE |
| 30921569262 |
mmtk |
clasp/macos/bytecode/no/no |
139 |
after FINALIZERS-GENERAL-REMOVE |
| 31721433418 |
fix/install-rpath |
clasp/ubuntu/native/yes/no |
139 |
compiling snapshot.lisp |
Two observations that seem worth more than the individual crashes:
It is overwhelmingly a macOS phenomenon — 4 of 5, across both clasp and cando, and across
both native and bytecode. Build mode is evidently not the variable.
Run 30921569262 crashed three cells simultaneously, all at the same site. Three independent
machines reaching the same point and dying together suggests something deterministic about the
state at that moment rather than a rare race.
Site A — compiling snapshot.lisp
Running snapshot suite...
; Compiling file: SYS:SRC;LISP;REGRESSION-TESTS;SNAPSHOT.LISP.NEWEST
; (IN-PACKAGE #:CLASP-TESTS)
FAILED: [code=132] test-boehmprecise
A healthy run echoes both (TEST ...) forms next, then Passed SLAD-SNAPSHOT /
Passed SLAD-EXECUTABLE. Here it dies after IN-PACKAGE and before the first test form is even
read — so this is not save-lisp-and-die; nothing has been saved or linked. It is a crash in
the compiler, on the first form of the file, which is #+use-precise-gc (test slad-snapshot ...).
Site B — immediately after FINALIZERS-GENERAL-REMOVE
Passed FINALIZERS-GENERAL
Passed FINALIZERS-GENERAL-REMOVE
FAILED: [code=139] test-boehmprecise
Where it probably is
Both sites are GC-adjacent: one is the compiler in a precise-GC build, the other lands immediately
after the finalizer tests. #1807 rewrote stop-the-world, root scanning and stack scanning, and
site B was observed on the mmtk branch itself. A collection arriving at the wrong moment fits
both, and the specific files are likely incidental — whatever allocates at the wrong instant would
do.
Reproduction — NOT achieved, 30 attempts
All on f236401f6 (main at 205f83f plus open PRs), boehmprecise, clean unpatched trees:
| host |
configuration |
runs |
crashes |
| Ubuntu 24.04, 16 core / 157 GB, clang-18 |
TEST_SUITES=snapshot only, incl. 2 cores via taskset |
12 |
0 |
| " |
full suite, full width |
8 |
0 |
| " |
full suite, 2 cores + GC_FREE_SPACE_DIVISOR=50 |
4 |
0 |
| macOS arm64, 10 core / 32 GB, LLVM 22 |
full suite, GC_FREE_SPACE_DIVISOR=50 |
6 |
0 |
Every one completed at 1997 successes with the same five expected failures.
Two things this rules out, which may save someone else the time: it is not provoked by
hammering the snapshot suite alone, and it is not simply a matter of raising GC frequency on a
large machine. The distinguishing factor is most plausibly the runners themselves — a GitHub macOS
runner is roughly 3 cores with far less memory than either box tested here. Constraining a local
machine to that shape is the obvious next experiment and has not been done.
Why it matters
ninja -C build testintermittently dies on a fatal signal — SIGSEGV or SIGILL — rather thanfailing a test. It happens on
mainand on themmtkbranch with no PR involved, at two distinctpoints in the suite, and it is currently the most common CI failure mode: across the last 30
failed
test.ymlruns,Run regression testswas the failing step 30 times (ANSI 13, koga 18combined, Build 7).
I could not reproduce it locally in 30 attempts (details at the end), so this is a report rather
than a patch. The occurrence data below is verified job by job.
How to recognise it
code=1Successes: NNNNsummarycode=132code=139A run that fails with no
Successes:line has crashed, not failed. That distinction matters,because these currently get retried as flaky infrastructure.
Confirmed occurrences
mainsnapshot.lispmmtkFINALIZERS-GENERAL-REMOVEmmtkFINALIZERS-GENERAL-REMOVEmmtkFINALIZERS-GENERAL-REMOVEfix/install-rpathsnapshot.lispTwo observations that seem worth more than the individual crashes:
It is overwhelmingly a macOS phenomenon — 4 of 5, across both
claspandcando, and acrossboth
nativeandbytecode. Build mode is evidently not the variable.Run 30921569262 crashed three cells simultaneously, all at the same site. Three independent
machines reaching the same point and dying together suggests something deterministic about the
state at that moment rather than a rare race.
Site A — compiling
snapshot.lispA healthy run echoes both
(TEST ...)forms next, thenPassed SLAD-SNAPSHOT/Passed SLAD-EXECUTABLE. Here it dies afterIN-PACKAGEand before the first test form is evenread — so this is not
save-lisp-and-die; nothing has been saved or linked. It is a crash inthe compiler, on the first form of the file, which is
#+use-precise-gc (test slad-snapshot ...).Site B — immediately after
FINALIZERS-GENERAL-REMOVEWhere it probably is
Both sites are GC-adjacent: one is the compiler in a precise-GC build, the other lands immediately
after the finalizer tests. #1807 rewrote stop-the-world, root scanning and stack scanning, and
site B was observed on the
mmtkbranch itself. A collection arriving at the wrong moment fitsboth, and the specific files are likely incidental — whatever allocates at the wrong instant would
do.
Reproduction — NOT achieved, 30 attempts
All on
f236401f6(mainat 205f83f plus open PRs), boehmprecise, clean unpatched trees:TEST_SUITES=snapshotonly, incl. 2 cores viatasksetGC_FREE_SPACE_DIVISOR=50GC_FREE_SPACE_DIVISOR=50Every one completed at 1997 successes with the same five expected failures.
Two things this rules out, which may save someone else the time: it is not provoked by
hammering the snapshot suite alone, and it is not simply a matter of raising GC frequency on a
large machine. The distinguishing factor is most plausibly the runners themselves — a GitHub macOS
runner is roughly 3 cores with far less memory than either box tested here. Constraining a local
machine to that shape is the obvious next experiment and has not been done.
Why it matters
occurrence costs someone a bisect against a change that did not cause it.
flakiness and simply retried.
ninja test's exit code unreliable as a merge gate, compounding ninja test exits 0 when tests fail to register: a broken test file silently removes its tests from the run #1824.