v1 · standalone benchmark release
Introducing FMG-Bench, the Faith & Moral Guidance Benchmark, for evaluating large language model behavior in theological triage, moral guidance, and pastoral-adjacent contexts.
Alex Chao · Fide AI · 2026
Release status: the research companion page remains on fideai.org, while benchmark code, dataset files, result summaries, and paper artifacts are maintained in the standalone FMG-Bench repository and dataset page.
Current status
FMG-Bench v1 is maintained as a standalone benchmark repository with code, dataset files, result summaries, paper artifacts, release caveats, and citation metadata.
Dataset
The Hugging Face dataset contains the open v1 benchmark corpus: 120 base scenarios with 37 perturbation variants for lightweight inspection and reuse.
Repository boundary
This page explains the research. The standalone FMG-Bench repo is the source of truth for implementation, data, reproducibility instructions, and paper source.
Evaluation artifact
The public package separates research claims, benchmark data, scoring code, result summaries, reproduction notes, and interpretation limits so readers can inspect what was tested and what should not be inferred.
Abstract
People increasingly ask large language models for counsel on questions of faith, doctrine, and pastoral care. These questions are not ordinary information requests: some ask about core Christian beliefs, some ask about real disagreement among faithful traditions, some require humility, and some are pastoral situations where safety and human referral matter more than theological completeness. We introduce FMG-Bench, the Faith & Moral Guidance Benchmark, a 120-scenario benchmark for theological triage and pastoral guidance in English-language Christian contexts.
FMG-Bench v1 evaluates 14 advanced models across 8,792 scored responses, comparing raw model behavior with three guided instruction settings. Placing models inside a structured harness improves over raw model behavior by +3.96 points on average, with all 14 models improving.
The largest domain gain is pastoral application (+6.62), and the most safety-critical gain is escalation appropriateness (+10.8), measuring whether systems recognize when pastoral, clinical, legal, emergency, or community support is needed. The guided settings also improve robustness (92.88 → 98.02 stability). Perspective comparison helps secondary doctrine but can be counterproductive when applied to primary doctrine or urgent pastoral situations.
The benchmark is a measurement tool, not an endorsement of AI systems as pastoral authorities.
Key findings
+3.96 pts
Average improvement
Guided default vs. raw model across all 14 models. Every model improved.
+6.62 pts
Pastoral application
Largest gains where safety, referral, and care boundaries matter most.
+7.36 pts
Embodiment / escalation
Guided system dramatically improves appropriate pastoral escalation behavior.
98.02%
Robustness stability
Up from 92.88% raw. Guidance dramatically reduces variance under prompt perturbation.
Guided improvement by triage level
Model explorer
Toggle conditions on and off to see how system layers change model behavior. Every model improved under the guided default condition.
Scores are averaged across all scenarios and triage levels. Human calibration remains an active validation step. Higher is better (0–100 scale).
Triage framework
The central question is not “did the model answer correctly?” but “did the model respond in the right kind of way for the kind of issue at stake?”
Scenario sampler
Each scenario includes expected behaviors, disallowed failure modes, and a failure tag explaining what went wrong.
Scoring dimensions
Theological Quality
+3.72 guided
Grounding & Evidence
+4.23 guided
Preference Fidelity
+2.97 guided
Comparative Honesty
+3.07 guided
Failure taxonomy
Top failure tags by raw-condition frequency. Guided conditions reduce most of these substantially.
Rates shown for raw model condition. Frequency is proportion of scored items where tag was applied.
Benchmark design
Corpus construction
120 base scenarios across primary doctrine (25), secondary doctrine (35), tertiary doctrine (30), and pastoral application (30). Each scenario includes triage metadata, doctrine loci, tradition scope, expected behaviors, disallowed failure modes, and scenario-specific score weights.
System conditions
Four conditions: raw model (no system prompt), guided default (bounded theological and pastoral system layer), preference configured (user tradition and preferences applied), and perspective compare (multi-tradition framing). All conditions use neutral terminology in publication materials.
Scoring protocol
LLM-as-judge scoring with a three-model panel. Each response scored on five dimensions: theological/pastoral quality, grounding and evidence, preference fidelity, comparative honesty, and escalation appropriateness. Judge summaries and failure tags are recorded.
Robustness testing
Perturbation variants test whether guidance remains stable under paraphrase, pressure, false premise, emotional intensity, and point-of-view shifts. Robustness measured as score stability (guided: 98.02%, raw: 92.88%).
Human calibration
Required before strong claims about judge validity or pastoral adequacy. Protocol supports reviewer role, tradition, confidence notes, and agreement reports by triage level, tradition scope, and score dimension. Results are provisional until calibration is complete.
Evaluation artifact
FMG-Bench is designed for faith-facing questions first, but the release also follows the discipline expected of public evaluation artifacts: readers should be able to find the tested scope, method, artifacts, and limits without treating the score as an endorsement.
Release artifacts
Interpretation limits
Benchmark scores are not theological authority, pastoral authority, or universal product approval. They are evidence about behavior under named versions, prompts, conditions, rubrics, and evaluation procedures. Human calibration remains necessary before making strong claims about judge validity or pastoral adequacy. FMG-Bench is maintained by Fide AI as an independent research benchmark. Results should not be interpreted as endorsement of any product, model, denomination, or pastoral decision.
Next research frontier
FMG-Bench v1 focuses on theological triage and pastoral-adjacent guidance. Future Fide AI work will extend evaluation toward human dignity, formation, anthropomorphic boundary-setting, relational substitution risk, and institutional deployment readiness.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。