This document describes the design of the C kernel and its boundary with the Forth layer.
The whole of Ash can be explained by three boundaries:
┌──────────────────────────────────────────┐
│ │
│ source ─────────────────▶ token │
│ │
╞═══════════ semantic boundary ════════════╡
│ │
│ outer interpreter ──────▶ xt │
│ │
╞═══════════ execution boundary ═══════════╡
│ │
│ threaded engine ────────▶ machine │
│ │
╞══════════ optimization boundary ═════════╡
│ │
│ JIT │
│ │
└──────────────────────────────────────────┘
Above the semantic boundary, text is only cut into tokens — no meaning is assigned. Meaning appears when the outer interpreter maps a token to an xt or a number. The execution boundary separates deciding what to run from running it. The optimization boundary is where the JIT replaces threaded dispatch with native code, without changing anything above it.
The JIT is part of Ash's essential scope, not an optional extension. Its position below the optimization boundary describes the architecture — what it is allowed to touch — not its priority. The boundary defers when native code appears, never whether: everything below it is designed so that swapping threaded dispatch for native code is a planned move, not a rewrite.
Ash targets the ANS Forth Core word set, with selected later-standard words (e.g. PARSE-NAME from Forth-2012) where they simplify the system. CATCH and THROW belong to the optional Exception word set, but Ash builds them anyway: ABORT and QUIT are Core words and need the same machinery.
Forth has almost no syntax. There is no lexer → parser → AST → compiler pipeline; the model is much closer to tokenizer → interpreter.
input
↓
tokenizer next_token() — cuts tokens, nothing else
↓
token
↓
interpreter interpret_token()
├─ found in dictionary → execute
└─ parses as a number → push onto data stack
For example, 1 2 + proceeds as:
"1" → number → push 1
"2" → number → push 2
"+" → dictionary lookup → execute +
The tokenizer holds no judgment. All meaning lives in the interpreter. When the system enters compilation state, the token stream is unchanged — only the interpreter's response to a token changes:
word found
├─ immediate → execute now
└─ normal → compile its xt into the current definition
This is the essential property of Forth: the parser barely changes, and state changes what interpretation means.
A step-by-step trace of these mechanisms — call order, stack states, patching sequences — lives in EXECUTION.md.
src/
ash.h — cell type, VM state, dictionary entry definitions
token.c — next_token() (input source → token)
interp.c — interpret_token(), outer interpreter loop
exec.c — inner interpreter (threaded execution)
dict.c — dictionary representation and find_word()
stack.c — data / return stacks
prims.c — primitive words forming the bootstrapping boundary
main.c — REPL and boot
forth/
core.fs — derived words, control structures (the Forth layer)
This mirrors the principle in the README: the C layer provides only what is necessary to make Forth exist.
In Ash, the differences between a cell, an xt, a code field, a parameter field, and the instruction pointer are the execution model. The C type system should keep those boundaries visible rather than erase them all into cell_t:
typedef struct vm vm_t;
typedef intptr_t cell_t; /* one cell */
typedef cell_t *xt_t; /* execution token: address of a code field */
typedef void (*code_t)(vm_t *vm, xt_t xt); /* code-field routine */This separation costs nothing now and pays off most once the JIT starts rewriting code fields.
One assumption is explicit rather than hidden: Ash stores code_t function pointers in dictionary cells, and ISO C does not promise that a function pointer fits an integer cell. The stated platform — POSIX, x86_64 first — does promise it, so Ash declares the constraint with a _Static_assert in ash.h instead of complicating the dictionary layout to avoid it. The constraint is not the problem; hiding it would be.
typedef struct {
const char *buf; /* input buffer */
cell_t len;
cell_t in; /* >IN: parse position within buf */
cell_t source_id; /* 0 = terminal, -1 = string (EVALUATE), else fileid */
} input_source_t;
struct vm {
cell_t *dsp, *rsp; /* data / return stack pointers */
xt_t *ip; /* instruction pointer (threaded code) */
uint8_t *here; /* dictionary allocation pointer */
cell_t state; /* 0 = interpret, -1 = compile */
cell_t base; /* number conversion radix, initially 10 */
input_source_t src[SOURCE_DEPTH]; /* input source stack */
cell_t src_depth; /* src[src_depth] is the active source */
};Decisions that matter here:
state,base, and>INare plain cells in the VM. The standard treats them as system data a program may access; keeping them as addressable cells means the Forth layer can touch them withSTATE @,BASE !, or>IN !, so words like[,],HEX, andPOSTPONEcan be written in Forth without extending the C kernel.- The dictionary is a single contiguous data space through which
hereadvances.HERE,,, andALLOTfall out naturally, and the layout matches the ANS data-space semantics. Allocating entries individually (e.g. with malloc) would fight the language later.
The standard's input source is not the terminal; it is an abstraction over terminal, string, and file, with nested input that can be saved and restored. The TIB is therefore not the essence of the VM — the source abstraction is.
The active source is the top of a small stack of input_source_t. The tokenizer always reads from the active source; nothing else in the system knows where text comes from.
SOURCEreturns the activebuf len;>INis the active source'sincell;SOURCE-IDdistinguishes terminal, string, and file.EVALUATEpushes a string source, runs the same outer interpreter loop, and pops it.REFILLasks the active source for its next line (terminal read, file read; false for a string).- Bootstrapping
core.fsis nothing special: push a file source at startup and run the ordinary outer loop until it is exhausted.
One outer loop, many sources. The interpreter does not change when the input does.
/* Cut one whitespace-delimited token from the active input source,
starting at its >IN. Returns a pointer into the source buffer
(no copy). NULL when the source is exhausted. */
const char *next_token(vm_t *vm, size_t *len);- Zero judgment: skip whitespace, delimit, return.
- Returning a pointer + length into the input buffer matches the standard's text-parsing model: the parse area begins at
>INand parsing advances it. Exported as a primitive, this isPARSE-NAME(a Forth-2012 word Ash adopts), and the Forth layer can buildWORDand friends on top of it.
void interpret_token(vm_t *vm, const char *tok, size_t len) {
dict_entry_t *w = find_word(vm, tok, len);
if (w) {
if (vm->state && !(w->flags & F_IMMEDIATE))
compile_cell(vm, (cell_t)xt(w)); /* append xt to current definition */
else
run_xt(vm, xt(w)); /* immediate, or interpreting */
} else if (parse_number(tok, len, &n)) { /* radix = BASE */
if (vm->state) {
compile_cell(vm, (cell_t)xt_lit); /* compile LIT n */
compile_cell(vm, n);
} else {
push(vm, n);
}
} else {
error(vm, "undefined word", tok, len);
}
}The outer loop is nothing more than:
while ((tok = next_token(vm, &len)))
interpret_token(vm, tok, len);State affects only the two branches inside interpret_token. The token stream is identical in both states.
A note on conformance: the standard specifies per-word interpretation semantics and compilation semantics — for a normal definition, the compilation semantics are to append its execution semantics to the current definition. The xt + IMMEDIATE flag scheme above is not those semantics; it is Ash's implementation model of them. Ash initially represents a word's compilation behavior as an execution token plus an immediate flag. If a word ever requires semantics this model cannot express, it is the model that changes, not the standard.
The dictionary is where every other concept — xt, code field, parameter field, >BODY, CREATE, DOES>, colon definitions, JIT — either connects or fails to. The physical layout of an entry:
┌───────────────┐
entry ──▶ │ link │ 1 cell → previous entry (0 terminates)
├───────────────┤
│ flags │ 1 byte IMMEDIATE | HIDDEN
│ name_len │ 1 byte
│ name… │ name_len bytes
│ padding │ → next cell boundary
├───────────────┤
│ does │ 1 cell does-code address, 0 if unused = xt[-1]
├───────────────┤
xt ─────▶ │ code │ 1 cell code_t = xt[0]
├───────────────┤
body ───▶ │ parameter │ … data field
│ field │
└───────────────┘
All the address arithmetic is fixed and trivial:
xt = &code
>BODY = xt + 1 cell
does = xt[-1]
Putting the does-code address in the parameter field looks attractive but conflicts with the standard. By the time DOES> applies — : constant create , does> @ ; — the data field already contains data, and the standard fixes >BODY to the data-field address assigned at CREATE time. Prepending the does-address would move the data; appending it would put it at a word-dependent offset the handler cannot find.
So the does-address lives in a fixed slot adjacent to the code field, at xt[-1]. The data field never moves, >BODY stays xt + 1 for every word, and — the property that matters most — the code field remains a single cell: the one and only slot that changes when a word's execution strategy changes. The cost is one cell per entry.
class code does body
primitive C routine 0 —
colon docol 0 xt … exit
create docreate 0 data field
constant docon 0 value
does> word dodoes thread data field (unmoved)
Runtime behavior of each code-field routine:
docreate: push body
docon: push body[0]
docol: rpush ip; ip = body
dodoes: push body; rpush ip; ip = does — data-field address, then DOES> semantics
DOES> itself is an immediate word that compiles (does>) into the defining word. When the defining word runs, (does>) patches the most recent definition — does = ip; code = dodoes — and exits; the thread after (does>) belongs to the created word, not the definer.
Switching a word's execution strategy means writing one cell — the code field — and no caller ever changes. A hot does-word is compiled by translating its thread and swapping code; does stays where the compiler can read it. The full contract is the JIT contract section below.
Ash does not build CATCH / THROW on setjmp / longjmp. It builds them on the Forth return stack.
This works because of one property the inner interpreter must guarantee: executing a colon definition never nests C stack frames. docol and exit move ip and the Forth return stack; the C-level inner loop neither recurses nor unwinds. (The EXECUTE primitive must respect this too: dispatching a colon word adjusts ip and returns to the same loop.) A non-local return is therefore pure stack surgery on the data and return stacks — something Forth can express itself.
The C kernel contributes only stack-pointer access: sp@ sp! rp@ rp!. Everything else is core.fs, in the shape of the standard's own reference implementation:
variable handler
: catch ( xt -- exception# | 0 )
sp@ >r handler @ >r rp@ handler !
execute
r> handler ! r> drop 0 ;
: throw ( n -- )
?dup if
handler @ rp! r> handler !
r> swap >r sp! drop r>
then ;QUIT empties the return stack (rp!) and re-enters the outer interpreter loop; ABORT empties the data stack and performs QUIT; an uncaught THROW falls back to ABORT's behavior. The standard's rationale describes CATCH/THROW as a non-local return similar to C's setjmp/longjmp — Ash takes that comparison literally and then refuses the C mechanism, because exception semantics belong above the machine boundary. A small C kernel. The rest is Forth — including the exceptions.
The JIT lives entirely below the optimization boundary.
Before JIT compilation, a colon definition points at docol, and its body is a thread of execution tokens:
[ docol ] [ dup-xt | *-xt | exit-xt ]
After JIT compilation, only the code field changes:
[ native ] [ original thread may remain ]
Callers always enter a word through its code field, so threaded and native words may coexist freely. A threaded word may call a native word. A native word may transfer control to a threaded word. Immediate words require no special treatment: they are ordinary words that happen to execute during compilation and may be translated by the same machinery.
The code field realizes the execution boundary; swapping it is the optimization boundary made physical. Changing the execution strategy below the code field must not change semantics above it.
The JIT may evolve in stages.
-
Template JIT. Translate each xt in a thread into a native dispatch sequence. The first goal is only to remove the inner interpreter's repeated dispatch. No Forth semantics are reimplemented.
-
Primitive inlining. Known primitives such as
dup,+,@, and!may be expanded directly into native instructions.litbecomes an immediate push.branchand0branchbecome native control-flow instructions. -
Inlining and specialization. Native words may later inline other words, specialize operations, or perform further optimization as long as the observable Forth semantics remain unchanged.
JIT-generated code must preserve one invariant:
Executing Forth code must not create nested C stack frames that are required for Forth control flow.
This constraint follows from Ash's exception model. CATCH and THROW perform non-local control transfer by restoring the Forth data and return stacks. They do not unwind the C stack. Therefore, native Forth calls must not depend on the host C call stack for their continuation.
A native word calling another Forth word must place its continuation on the Forth return stack and transfer control without creating a persistent C frame:
native word
│
├─ push continuation onto Forth return stack
│
└─ jump to callee
The same rule applies to:
native → native
native → threaded
threaded → native
threaded → threaded
All four combinations use the same Forth control-flow model.
One decision this forces: the return stack needs a uniform continuation representation that both exit paths understand. exit pops a value and resumes it as a thread ip; a native word's continuation is a machine-code address. Either every continuation stays a thread ip — a native caller pushes a pointer to a one-cell trampoline thread whose xt re-enters the native code — or the exit protocol becomes an indirect jump through the popped value for all words. The trampoline form is currently favored: the threaded engine, the NULL-ip sentinel, and exit remain untouched. The contract requires only that one representation serve all four combinations.
This keeps CATCH and THROW independent of the JIT. Restoring the Forth return stack is sufficient to discard both threaded and native continuations.
The JIT must adapt to the exception model; the exception model must not become aware of the JIT.
JIT-generated code participates in the same execution boundary as every other word. The code field remains the single point through which a word's execution strategy is selected.
The exact native calling convention is architecture-specific, but it must preserve:
- the data-stack pointer
- the Forth return-stack pointer
- the instruction / continuation state
- the VM state required by Forth semantics
- the no-nested-C-frames invariant
- full stack materialization at every point where control may leave
the word:
THROWrestores stack pointers and discards everything else, so Forth state cached only in registers across a call into another word would be lost
The initial implementation targets x86_64.
Hotness metadata is not part of a Forth word's semantic representation. Execution counters and profiling information therefore live outside the dictionary entry, keyed by xt or another stable word identity. This keeps the dictionary layout independent of optimization policy.
A word becomes eligible for JIT compilation when the estimated cost of continuing threaded execution exceeds the cost of translating it. The initial implementation may use a simple invocation counter; more advanced policies may later include thread length, backedge counts, or measured execution cost.
Hotness is an optimization policy, not part of the language.
The C kernel contains only the primitive operations needed to bootstrap the Forth layer.
Words are not in C because Forth cannot express them — many of them could be defined in terms of one another. They are in C because they sit on the bootstrapping boundary Ash chose. The kernel splits them into two groups:
machine primitives — the boundary to the machine
stack manipulation dup drop swap over rot >r r> r@ depth
stack pointer access sp@ sp! rp@ rp!
arithmetic / logic + - * / mod = < > 0= and or xor invert
memory access @ ! c@ c!
I/O key emit type
threaded dispatch lit branch 0branch exit execute
bootstrap primitives — the minimum needed for Forth to define Forth
dictionary creation create here allot , find
compilation : ; immediate (does>)
parsing / state parse-name refill state >in base source-id
Everything above this boundary is derived Forth, written in core.fs:
derived Forth
2dup 2drop negate abs
if else then
begin until
do loop
constant variable does>
catch throw abort quit
...
Control structures are not primitives: they are immediate words over branch / 0branch. Exceptions are not C: they are return-stack surgery over sp@ sp! rp@ rp!. The project's ambition is to keep moving this boundary downward — to see how much of the C can be burned away, leaving ash.