Can LLMs decide amongst themselves who is the best?
Fork note: This is a fork of Language Model Council by Justin Zhao et al., extended by Lorenzo Vigentini.
- Recording: https://youtu.be/hI0XCE27QqE
- Slides: https://bit.ly/44XSEnh
Language models are outpacing our abilities to evaluate them. Yet, as capable as we hype up LLMs to be, all of the most popular evals the world uses to rank LLMs are still exclusively human-curated and human-designed.
When a new model gets released, Twitter and Reddit get flooded with claims about this latest model being the new best at one thing or another. With so many benchmarks out there, deciding which model is the best has become a matter of taste. I became curious to answer a bit of a different question: Can we get LLMs to decide amongst themselves who is the best?
LMSYS demonstrated that GPT-4 agrees with humans at roughly the same rate that humans agree with each other. Today, more and more of us are using models like GPT-4 in place of human raters. But GPT-4 is only one model and today, it's actually one of the "weakest" models on Chatbot Arena.
And how much does a model’s Chatbot Arena rank really tell us about broad human alignment? The PRISM dataset — which was recognized as the Best Paper at NeurIPS in 2024 — showed that rankings shift dramatically depending on which humans you sample. It turns out that for open-ended prompts, there's no single "right" answer — and different populations have fundamentally different views of what “best” means.
On the other side of coin, researchers have also found that LLMs exhibit all sorts of biases on gender, religion, and even value of life. More recent work from the Societal Impacts team at Anthropic and Researchers at the Center for AI Safety are finding even more evidence that: Each model carries its own values — inherited unintentionally or intentionally — from the societies and organizations that built them, and the humans that curate and choose their training data.
In the future, you can imagine that we'll have strong LLMs from every country and many organizations — each shaped by different geopolitical priorities, specializations, and cultural values. More of us are coming to rely on AI for more things, and for things like advising on policies, giving life advice, predicting the future, these domains may be too speculative or too subjective for any single model to evaluate fairly, and disagreement between AI systems will become more prevalent.
So how do we make decisions amidst dissenting opinions in human society? Well, one thing we do in America is called democracy. We all know democracy is far from perfect, but at its core, democracy is a profound idea based on decentralizing power, giving everyone a voice, and relying on the collective to make an important decision.
That's the spirit behind the Language Model Council. Put LLMs in a democracy and give them agency so that they can elect a leader amongst themselves.
This library provides an open-source archive of all the code and analysis used in our original research paper, and serves as a tool for anyone interested in using a council of LLMs to self-evaluate on a set of prompts.
- Install with pip.
pip install ilm-council- Add your openrouter secrets to a
.envfile.
OPENROUTER_API_KEY = ""See .env.example for an example. Check here for your openrouter API key.
- Configure and execute your council.
You can run the council as a standalone python script or in jupyter notebooks. See examples/ for example notebooks.
from ilm_council import LanguageModelCouncil
from dotenv import load_dotenv
def main():
load_dotenv()
lmc = LanguageModelCouncil(
models=[
"deepseek/deepseek-r1-0528",
"google/gemini-2.5-flash-lite-preview-06-17",
"x-ai/grok-3-mini",
"meta-llama/llama-3.1-8b-instruct",
],
)
# Run the council on any prompt of your choosing.
completion, judgment = await lmc.execute("Say hello.")
# Run the council on many prompts in parallel.
completions, judgements = await lmc.execute(
["Say hello.", "Say goodbye.", "What is your name?", "What is 1 + 1?"]
)
# Save and load your council.
lmc.save("run_0")
lmc.load("run_0")
# Shows a leaderboard and returns a scores dataframe.
return lmc.leaderboard()
asyncio.run(main())This fork extends the original library with several new capabilities for more sophisticated council composition and evaluation.
Full documentation: See docs/ROLES.md for complete details, system prompts, and behavioral examples.
The role system provides two orthogonal dimensions for council composition:
| Dimension | Question | Mechanism |
|---|---|---|
| Persona | How should this judge evaluate? | System prompt injection |
| Model Affinity | Which model should judge this task? | Model selection based on capabilities |
Personas are behavioral overlays that shape how a model evaluates responses. Each persona has a specific evaluation focus and strictness level.
| Persona | Focus | Strictness | Purpose |
|---|---|---|---|
critical_reviewer |
accuracy, logic, completeness | 4/5 | Find weaknesses and gaps |
creative_advocate |
originality, creativity, engagement | 2/5 | Champion novel approaches |
user_advocate |
clarity, usefulness, accessibility | 3/5 | Represent end-user perspective |
domain_expert |
accuracy, depth, expertise | 4/5 | Verify technical correctness |
devil_advocate |
alternative perspectives, bias | 3/5 | Challenge consensus |
synthesizer |
balance, nuance, integration | 3/5 | Find common ground |
code_reviewer |
correctness, security, readability | 4/5 | Professional code review |
security_reviewer |
vulnerabilities, attack surface | 5/5 | Security-first evaluation |
fact_checker |
factual accuracy, citations | 5/5 | Verify claims |
writing_editor |
clarity, structure, grammar | 3/5 | Evaluate prose quality |
from ilm_council.roles import PRESET_ROLES
# Get a persona and examine its system prompt
critic = PRESET_ROLES.get_persona("critical_reviewer")
print(critic.system_prompt)
# "You are a critical reviewer on an evaluation council. Your role is to:
# - Identify weaknesses, gaps, and logical flaws in responses
# - Challenge assumptions and unsupported claims..."Model affinity roles map capabilities to models known to excel at those tasks:
| Role | Preferred Models | Use Case |
|---|---|---|
code_expert |
DeepSeek-Coder, Claude | Code generation, review |
reasoning_expert |
o1, R1 | Math, logic, complex problems |
creative_writer |
Claude, GPT-4o | Creative writing, style |
fast_responder |
Gemini Flash, Haiku | Quick, cost-effective |
frontier_model |
Claude Sonnet 4, o1 | Highest capability |
code_expert = PRESET_ROLES.get_model_role("code_expert")
print(code_expert.preferred_models)
# ['deepseek/deepseek-coder', 'anthropic/claude-3.5-sonnet', ...]The same response evaluated by different personas yields different scores:
| Persona | Score | Reasoning |
|---|---|---|
| critical_reviewer | 6/10 | "Has gaps, doesn't explain why, analogy imprecise" |
| creative_advocate | 8/10 | "Love the mirror analogy! Creative and memorable" |
| domain_expert | 5/10 | "Correct but shallow, missing technical depth" |
See docs/ROLES.md for the full example and all system prompts.
Create councils from templates or compose them programmatically.
from ilm_council.formation import compose_council, DEFAULT_COMPOSER
# List available templates
templates = DEFAULT_COMPOSER.list_templates()
# ['code_review_council', 'creative_writing_council', 'reasoning_council',
# 'fact_checking_council', 'balanced_council', 'brainstorm_council',
# 'debate_council', 'quick_council', 'frontier_council', 'safety_council',
# 'long_document_council', 'multilingual_council']
# Create from template
formation = compose_council(
template="code_review_council",
available_models=[
"anthropic/claude-3.5-sonnet",
"openai/gpt-4o",
"deepseek/deepseek-coder",
]
)
print(f"Council: {formation.name}, {formation.size} members")
for member in formation.members:
persona = member.persona.name if member.persona else "none"
print(f" - {member.model}: {persona}")
# Task-driven composition (auto-selects template)
formation = compose_council(
task_type="coding", # -> code_review_council
available_models=[...]
)
# Custom composition with explicit personas
formation = compose_council(
models=["anthropic/claude-3.5-sonnet", "openai/gpt-4o", "deepseek/deepseek-coder"],
personas=["critical_reviewer", "user_advocate", "synthesizer"]
)Automatically classify tasks and route them to appropriate councils.
from ilm_council.classification import classify_task, route_to_formation
# Classify a prompt
classification = classify_task("Review this Python code for security vulnerabilities")
print(classification.task_type) # TaskType.CODING
print(classification.complexity) # ComplexityLevel.MODERATE
print(classification.stakes) # StakesLevel.MEDIUM
print(classification.confidence) # 0.75
# Route classification to formation
formation = route_to_formation(
classification,
available_models=["anthropic/claude-3.5-sonnet", "openai/gpt-4o"]
)
# Returns code_review_council for CODING tasks
# Classification routing rules:
# - CODING -> code_review_council
# - REASONING -> reasoning_council
# - CREATIVE -> creative_writing_council
# - FACTUAL -> fact_checking_council
# - SAFETY -> safety_council
# - CRITICAL stakes -> frontier_council (best models)
# - LOW stakes -> quick_council (fast models)Expose the council as an MCP (Model Context Protocol) server for use by Claude and other LLMs.
# Start the MCP server
python -m ilm_council.mcp.serverAvailable MCP tools:
| Tool | Description |
|---|---|
list_templates() |
List available formation templates |
create_council() |
Create council from models or template |
classify_prompt() |
Classify a task for optimal routing |
classify_and_form_council() |
Classify and form council in one step |
get_session() |
Get session state |
list_sessions() |
List all active sessions |
delete_session() |
Delete a session |
council_info() |
Get server information |
When an LLM invokes the council via MCP, it acts as the "head of council" - orchestrating the evaluation process and synthesizing results.
# Example: Claude invoking via MCP tools
result = create_council(
template="code_review_council",
models=["anthropic/claude-3.5-sonnet", "openai/gpt-4o"]
)
session_id = result["session_id"]
# Or auto-classify and form
result = classify_and_form_council(
"Debug this recursive function that's causing a stack overflow",
available_models=["anthropic/claude-3.5-sonnet", "deepseek/deepseek-coder"]
)Following research best practices (Karpathy's llm-council), responses are anonymized during judging to prevent model bias.
from ilm_council.mcp import ResponseAnonymizer
anon = ResponseAnonymizer()
anon.add_response("anthropic/claude-3.5-sonnet", "Hello from Claude")
anon.add_response("openai/gpt-4o", "Hello from GPT")
anon.add_response("deepseek/deepseek-coder", "Hello from DeepSeek")
# During judging - models see anonymous labels
for label, response in anon.get_anonymized():
print(f"{label}: {response}")
# Response A: Hello from DeepSeek
# Response B: Hello from Claude
# Response C: Hello from GPT
# After consensus - reveal the mapping
print(anon.reveal())
# {'Response A': 'deepseek/deepseek-coder', 'Response B': 'anthropic/claude-3.5-sonnet', ...}Our paper, "Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks", focuses on a case study involving 20 large language models (LLMs) to evaluate each other on a highly subjective emotional intelligence task, and was the first to study the application of LLM-as-a-Judge in a democratic setting.
Our paper was accepted to NAACL Main, and was presented in Alberqueque, New Mexico in May 2025. You can watch the recording of the talk on YouTube. Slides can be found here.
Authors:
- Justin Zhao (Independent -> Research Engineer @ Meta Superintelligence Labs)
- Flor Miriam Plaza-del-Arco (Researcher @ Bocconi University -> Assistant Professor @ Leiden University)
- Benjamin Genchel (ML Engineer @ Spotify -> Independent)
- Amanda Cercas Curry (Researcher @ Bocconi University -> Research Scientist @ CENTAI)
You can find in-depth jupyter notebooks to reproduce the findings and figures reported in the
Language Model Council paper under analysis/.
- Website: https://lilm-council.com
- Dataset: https://huggingface.co/datasets/lilm-council/emotional_application
- Paper: https://arxiv.org/abs/2406.08598
- Recording: https://youtu.be/hI0XCE27QqE
- Slides: https://bit.ly/44XSEnh
For ease of maintenance, all model outputs are served by OpenRouter. The original implementation used for the paper used each organization's custom API endpoint through REST, which resulted in a lot of boilerplate code to manage different REST API request schemas and response formats. OpenRouter solves this for us by enabling us to query more models under a single unified interface. For maximally parallelized model querying, we adhere to OpenRouter's rate limits, which we fetch using your API key before the first batch of requests.
If you find this work helpful or interesting, please consider citing it as so:
@inproceedings{zhao-etal-2025-language,
title = {Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks},
author = {Zhao, Justin and Plaza-del-Arco, Flor Miriam and Genchel, Benjamin and Curry, Amanda Cercas},
editor = {Chiruzzo, Luis and Ritter, Alan and Wang, Lu},
booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
pages = {12395--12450},
address = {Albuquerque, New Mexico},
month = apr,
year = {2025},
publisher = {Association for Computational Linguistics},
doi = {10.18653/v1/2025.naacl-long.617},
url = {https://aclanthology.org/2025.naacl-long.617/},
}