SAEVerbalizer

Generating Explanations for Sparse Autoencoder Features
via Representation Verbalization

Paper · PDF · Verbalizer · Adapter · Data

SAEVerbalizer generates natural-language explanations of Sparse Autoencoder (SAE) features directly from their decoder directions. It injects a feature direction into an LLM's hidden states and trains the downstream layers to verbalize the resulting internal representation. At inference time, unseen features can be explained without retrieving per-feature activation examples.

The learned verbalization capability also transfers across separately trained SAEs. A lightweight representation-space adapter further extends it to SAE features from different language models. See the paper for full details.

This repository provides inference and Reference Agreement (RA) evaluation for the paper's 27B verbalizer and its 1B-to-27B representation adapter.

Resources

Resource	Link
27B verbalizer	THU-KEG/SAEVerbalizer-27B
1B-to-27B adapter	THU-KEG/SAEVerbalizer-Adapter-1B-to-27B
Evaluation data	THU-KEG/SAEVerbalizer-Data
27B source SAE	Gemma Scope 2, layer 16, width 262k, l0_medium
1B source SAE	Gemma Scope 2, layer 7, width 262k, l0_medium
RA judge	Qwen3-30B-A3B-Instruct-2507
Backbone	Gemma-3-27B-IT

Model weights and datasets are hosted on Hugging Face rather than committed to this repository.

Evaluation sets

The paper uses three mutually disjoint feature sets. Each example contains an SAE feature ID and its reference natural-language explanation from Neuronpedia. The 27B verbalizer is evaluated on layer-16 features from the 27B SAE, while the 1B-to-27B adapter-based verbalization system is evaluated on layer-7 features from the 1B SAE. Both collections follow the same split definitions and selection criteria.

Set	Abbreviation	Samples	Selection
Global Train-Standard	GTS	1,000	Global sample passing the training qualification standard
Low-Index Gold	LIG	200	Low-index sample passing the stricter gold standard
Global Gold	GG	1,000	Global sample passing the stricter gold standard

The data repository uses this layout:

evaluation/                         # 27B layer-16 features
├── global_train_standard_1000.json
├── low_index_gold_200.json
└── global_gold_1000.json
adapter/1b_to_27b/evaluation/       # 1B layer-7 features
├── global_train_standard_1000.json
├── low_index_gold_200.json
└── global_gold_1000.json

Quick start

1. Install inference dependencies

python -m venv .venv-inference
source .venv-inference/bin/activate
pip install --upgrade pip
pip install -r requirements-inference.txt

2. Download the artifacts

Log in to Hugging Face first if any gated resource requires authentication.

hf auth login

hf download THU-KEG/SAEVerbalizer-27B \
  --local-dir checkpoints/SAEVerbalizer-27B

hf download google/gemma-scope-2-27b-it \
  resid_post/layer_16_width_262k_l0_medium/params.safetensors \
  --local-dir sae/gemma-scope-2-27b-it

hf download THU-KEG/SAEVerbalizer-Data \
  --repo-type dataset \
  --include "evaluation/*.json" \
  --local-dir data

3. Run inference

export CHECKPOINT_PATH=checkpoints/SAEVerbalizer-27B
export SAE_PATH=sae/gemma-scope-2-27b-it/resid_post/layer_16_width_262k_l0_medium/params.safetensors

CUDA_VISIBLE_DEVICES=0 bash run_inference.sh

Predictions are saved to outputs/predictions/predictions.json. For a quick smoke test, evaluate two examples from each split:

CUDA_VISIBLE_DEVICES=0 bash run_inference.sh --limit 2

4. Run Reference Agreement evaluation

Use a separate environment for vLLM and the judge model:

deactivate
python -m venv .venv-judge
source .venv-judge/bin/activate
pip install --upgrade pip
pip install -r requirements-judge.txt

export JUDGE_MODEL_PATH=Qwen/Qwen3-30B-A3B-Instruct-2507
CUDA_VISIBLE_DEVICES=0 bash run_judge.sh

Per-example judgments and aggregate RA scores are saved to outputs/judged.json. RA measures the proportion of generated explanations that the independent judge considers semantically consistent with their reference explanations. Following the paper, judging uses Qwen3-30B-A3B-Instruct-2507 with temperature 0.

1B-to-27B adapter

The linear adapter maps a decoder direction from the Gemma Scope 2 1B layer-7 SAE into the hidden space of the 27B verbalizer at layer 16. The mapped direction is normalized and passed to the same verbalization procedure as the main experiment.

Download the adapter and its source SAE:

hf download THU-KEG/SAEVerbalizer-Adapter-1B-to-27B \
  --local-dir checkpoints/SAEVerbalizer-Adapter-1B-to-27B

hf download google/gemma-scope-2-1b-it \
  resid_post/layer_7_width_262k_l0_medium/params.safetensors \
  --local-dir sae/gemma-scope-2-1b-it

hf download THU-KEG/SAEVerbalizer-Data \
  --repo-type dataset \
  --include "adapter/1b_to_27b/evaluation/*.json" \
  --local-dir data

Run adapter inference with:

export VERBALIZER_PATH=checkpoints/SAEVerbalizer-27B
export ADAPTER_PATH=checkpoints/SAEVerbalizer-Adapter-1B-to-27B/adapter.safetensors
export SOURCE_SAE_PATH=sae/gemma-scope-2-1b-it/resid_post/layer_7_width_262k_l0_medium/params.safetensors

CUDA_VISIBLE_DEVICES=0 bash run_adapter_inference.sh

This writes outputs/adapter_predictions/predictions.json. The same judge is used for both inference paths:

export JUDGE_MODEL_PATH=Qwen/Qwen3-30B-A3B-Instruct-2507
export PREDICTIONS_PATH=outputs/adapter_predictions/predictions.json
export JUDGED_OUTPUT_PATH=outputs/adapter_judged.json

CUDA_VISIBLE_DEVICES=0 bash run_judge.sh

For a quick pipeline check, append --limit 2 to the adapter inference command. The adapter artifact contains adapter.safetensors and adapter_config.json; the latter fixes the source and target dimensions and layers used by adapter_inference.py.

Inference configuration

The released setup follows the paper's 27B-L16 configuration:

Setting	Value
Backbone	Gemma-3-27B-IT
SAE	Gemma Scope 2, residual stream, layer 16, width 262k, l0_medium
Direction	Normalized SAE decoder vector
Intervention	Norm-matched additive injection
Injection layer	16
Injection strength	0.2
Injection span	Final four prompt tokens
Decoding	Greedy, up to 15 new tokens

The adapter setup keeps the same prompt, target injection, and decoding settings. It maps layer-7, width-262k Gemma Scope 2 1B decoder directions through a learned linear projection from 1,152 to 5,376 dimensions. The resulting mapped direction is normalized before injection. Only the projection matrix is applied (Wv); the saved affine bias is not part of a direction transformation. The projection is cast to bfloat16 before mapping, matching the paper evaluation.

Hardware and scope

Inference and judging were tested separately on one NVIDIA A100 80GB GPU. device_map="auto" can distribute the 27B verbalizer across multiple visible GPUs. The released checkpoint targets the SAE and prompt configuration above; other layers, widths, or model families require separate validation or the adapters described in the paper.

Repository structure

SAEVerbalizer/
├── .gitignore
├── adapter_inference.py         # 1B-to-27B adapter inference
├── inference.py                 # 27B verbalizer inference
├── judge.py                     # shared RA evaluation
├── LICENSE
├── run_adapter_inference.sh
├── run_inference.sh             # inference entry point
├── run_judge.sh                 # evaluation entry point
├── requirements-inference.txt
├── requirements-judge.txt
└── README.md

Downloaded artifacts and generated outputs are placed in Git-ignored checkpoints/, sae/, data/, and outputs/ directories.

Release roadmap

• 27B verbalizer inference and RA evaluation
• GTS, LIG, and GG evaluation sets
• Full evaluation reproduction
• Complete 27B verbalizer checkpoint upload
• 27B verbalizer training code and data
• 1B-to-27B adapter inference
• 1B-to-27B adapter checkpoint and evaluation data
• Adapter training code and data
• Data selection and split-construction code

Citation

If you use SAEVerbalizer, please cite:

@article{meng2026saeverbalizer,
  title   = {SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization},
  author  = {Meng, Weihan and Guo, Hongzhu and Jing, Yi and Liu, Dewen and Yao, Zijun and Wang, Xiaozhi and Hou, Lei and Li, Juanzi},
  journal = {arXiv preprint arXiv:2608.13538},
  year    = {2026}
}

License

The code in this repository is released under the MIT License. Model weights and datasets are distributed separately under their respective terms.