If you're running LLM outputs through n8n workflows, you probably have no systematic way to verify what the model actually produced.
Did it hallucinate? Did it stay grounded in your source data? Was the answer accurate?
We just launched a native n8n node for RagMetrics that solves this.
How it works
3 nodes and you're evaluating every AI output in your workflow:
Trigger → Edit Fields → RagMetrics Evaluation
The RagMetrics node accepts:
question — the original user query
answer — the model-generated response
ground_truth — the correct expected answer
context — source documents for grounding evaluation
conversation — session ID for grouping evaluations
evaluation_group — your RagMetrics criteria configuration
And returns structured JSON with:
Criteria name (Accuracy, Hallucination, Grounding etc)
Score 1–5
Detailed reasoning for the score
Token usage for cost tracking
What you can do with the score
Once you have a score in your workflow you can:
→ Route outputs below a threshold to a human review queue
→ Trigger Slack or email alerts when hallucination is detected
→ Log every evaluation to your RagMetrics dashboard automatically
→ Block downstream actions when quality is too low
Two evaluation methods
Live AI Evaluation — uses a pre-configured Evaluation Group for consistent scoring across multiple evaluations. Ideal for production monitoring and batch processing.
Direct Evaluation API — submit single question-answer pairs for immediate scoring without an Evaluation Group. Perfect for ad-hoc evaluations and quick testing.
Quick setup
Create a RagMetrics account at ragmetrics.ai
Configure your judge model API key in the dashboard
Create an Evaluation Group and select your criteria
Add your RagMetrics API key to n8n credentials
Add the RagMetrics Evaluation node to your workflow
Map your fields and connect to downstream logic
Get started
📄 Node documentation: ragmetrics.ai/resources/n8n-node
⚡ Starter workflow ready to import: ragmetrics.ai/resources/n8n-simple-workflow
Questions or want help getting set up:
📧 olivier@ragmetrics.ai
📱 +1 917 767 4075