Synthesize and compare multiple LLM responses with OpenRouter council

Description

This template adapts Andrej Karpathy’s LLM Council concept for use in n8n, creating a workflow that collects, evaluates, and synthesizes multiple large language model (LLM) responses to reduce individual model bias and improve answer quality.

🎯 The gist

This LLM Council workflow acts as a moderation board for multiple LLM “opinions”:

The same question is answered independently by several models.
All answers are anonymized.
Each model then evaluates and ranks all responses.
A designated Council Chairman model synthesizes a final verdict based on these evaluations.
The final output includes:
The original query
The Chairman’s verdict
The ranking of each response by each model
The original responses from all models

The goal is to reduce single‑model bias and arrive at more balanced, objective answers.

🧰 Use cases

This workflow enables several practical applications:

Receiving more balanced answers by combining multiple model perspectives
Benchmarking and comparing LLM responses
Exploring diverse viewpoints on complex or controversial questions

⚙️ How it works

The workflow leverages OpenRouter, allowing access to many LLMs through a single API credential.
In the Initialization node, you define:
Council member models: Models that answer the query and later evaluate all responses
Chairman model: The model responsible for synthesizing the final verdict
Any OpenRouter-supported model can be used: https://openrouter.ai/models
For simplicity:
Input is provided via a Chat Input trigger
Output is sent via an email node with a structured summary of the council’s results

👷 How to use

Select the LLMs to include in your council:
Council member models: Models that independently answer and evaluate the query. The default template uses:
openai/gpt-4o
google/gemini-2.5-flash
anthropic/claude-sonnet-4.5
perplexity/sonar-pro-search
Chairman model: Choose a model with a sufficiently large context window to process all evaluations and rankings.
Start the Chat Input trigger.
Observe the workflow execution and review the synthesized result in your chosen output channel.

⚠️ Avoid using too many models simultaneously. The total context size grows quickly (n responses + n² evaluations), which may exceed the Chairman model’s context window.

🚦 Requirements

OpenRouter API access** configured in n8n credentials
SMTP credentials** for sending the final council output by email (or replace with another output method)

🤡 Customizing this workflow

Replace the Chat Input trigger with alternatives such as Telegram, email, or WhatsApp.
Redirect output to other channels instead of email.
Modify council member and chairman models directly in the Initialization node by updating their OpenRouter model names.