Skip to content

Regression suite dies on fatal signals (SIGSEGV/SIGILL), mostly on macOS, at two GC-adjacent points #1837

Description

@dg1sbg

ninja -C build test intermittently dies on a fatal signal — SIGSEGV or SIGILL — rather than
failing a test. It happens on main and on the mmtk branch with no PR involved, at two distinct
points in the suite, and it is currently the most common CI failure mode: across the last 30
failed test.yml runs, Run regression tests was the failing step 30 times (ANSI 13, koga 18
combined, Build 7).

I could not reproduce it locally in 30 attempts (details at the end), so this is a report rather
than a patch. The occurrence data below is verified job by job.

How to recognise it

exit meaning tell
code=1 ordinary test failure prints a Successes: NNNN summary
code=132 SIGILL (128+4) no summary — the process died
code=139 SIGSEGV (128+11) no summary

A run that fails with no Successes: line has crashed, not failed. That distinction matters,
because these currently get retried as flaky infrastructure.

Confirmed occurrences

run branch cell signal crash site
31214144149 main clasp/macos/native/yes/no 132 compiling snapshot.lisp
30921569262 mmtk clasp/macos/native/yes/no 139 after FINALIZERS-GENERAL-REMOVE
30921569262 mmtk cando/macos/native/yes/no 139 after FINALIZERS-GENERAL-REMOVE
30921569262 mmtk clasp/macos/bytecode/no/no 139 after FINALIZERS-GENERAL-REMOVE
31721433418 fix/install-rpath clasp/ubuntu/native/yes/no 139 compiling snapshot.lisp

Two observations that seem worth more than the individual crashes:

It is overwhelmingly a macOS phenomenon — 4 of 5, across both clasp and cando, and across
both native and bytecode. Build mode is evidently not the variable.

Run 30921569262 crashed three cells simultaneously, all at the same site. Three independent
machines reaching the same point and dying together suggests something deterministic about the
state at that moment rather than a rare race.

Site A — compiling snapshot.lisp

Running snapshot suite...
; Compiling file: SYS:SRC;LISP;REGRESSION-TESTS;SNAPSHOT.LISP.NEWEST
;   (IN-PACKAGE #:CLASP-TESTS)
FAILED: [code=132] test-boehmprecise

A healthy run echoes both (TEST ...) forms next, then Passed SLAD-SNAPSHOT /
Passed SLAD-EXECUTABLE. Here it dies after IN-PACKAGE and before the first test form is even
read
— so this is not save-lisp-and-die; nothing has been saved or linked. It is a crash in
the compiler, on the first form of the file, which is #+use-precise-gc (test slad-snapshot ...).

Site B — immediately after FINALIZERS-GENERAL-REMOVE

Passed FINALIZERS-GENERAL
Passed FINALIZERS-GENERAL-REMOVE
FAILED: [code=139] test-boehmprecise

Where it probably is

Both sites are GC-adjacent: one is the compiler in a precise-GC build, the other lands immediately
after the finalizer tests. #1807 rewrote stop-the-world, root scanning and stack scanning, and
site B was observed on the mmtk branch itself. A collection arriving at the wrong moment fits
both, and the specific files are likely incidental — whatever allocates at the wrong instant would
do.

Reproduction — NOT achieved, 30 attempts

All on f236401f6 (main at 205f83f plus open PRs), boehmprecise, clean unpatched trees:

host configuration runs crashes
Ubuntu 24.04, 16 core / 157 GB, clang-18 TEST_SUITES=snapshot only, incl. 2 cores via taskset 12 0
" full suite, full width 8 0
" full suite, 2 cores + GC_FREE_SPACE_DIVISOR=50 4 0
macOS arm64, 10 core / 32 GB, LLVM 22 full suite, GC_FREE_SPACE_DIVISOR=50 6 0

Every one completed at 1997 successes with the same five expected failures.

Two things this rules out, which may save someone else the time: it is not provoked by
hammering the snapshot suite alone, and it is not simply a matter of raising GC frequency on a
large machine. The distinguishing factor is most plausibly the runners themselves — a GitHub macOS
runner is roughly 3 cores with far less memory than either box tested here. Constraining a local
machine to that shape is the obvious next experiment and has not been done.

Why it matters

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions