Ask ChatGPT a real theology question — something like "what does Paul mean by justification in Romans 5" — and you'll usually get an answer that sounds like it was written by a nervous seminary intern. It hedges. It lists three views without picking one. It cites nothing. And if you ask it a follow-up question, it will happily contradict itself.
Pastors tell me some version of this story constantly. They don't conclude "AI is dumb." They conclude "AI can't be trusted with Scripture." Both conclusions are wrong, and understanding why ChatGPT gives weak Bible answers requires looking past the output and into the architecture that produced it.
This isn't a knock on OpenAI's engineering. It's a structural fact about what a general-purpose large language model is — and why that structure is a bad fit for theological research specifically. Once you see the architecture, the hedging stops being mysterious. It becomes predictable.
An LLM Predicts. It Doesn't Look Up.
Here's the part that gets skipped in most explanations: a large language model doesn't "know" the Bible the way a concordance knows it. It was trained to predict the next most probable word, given everything that came before it, based on patterns learned from a massive scrape of internet text — forums, blogs, commentaries, sermons, Wikipedia, academic papers, all of it mixed together with no tagging for which claims are reliable.
When you ask it about Melchizedek, it isn't retrieving a citation. It's generating the statistically likely continuation of "Melchizedek is..." based on the blended average of everything it read about Melchizedek during training. Reinforcement learning from human feedback (RLHF) — the process that makes these models sound agreeable and safe — layers on top of that a strong incentive to hedge on anything contested, because hedging is what got rewarded during training.
That's not retrieval. That's not even blending traditions in any principled way. It's invention dressed up as an answer — a smoothie made from whatever was in the training set, poured out in confident-sounding prose.
Retrieval-augmented generation (RAG), in plain terms, is the fix: instead of letting the model answer purely from what it memorized during training, you first pull the actual source documents relevant to the question — a Bible passage, a lexicon entry, a scholar's commentary — and hand those to the model as context before it writes anything. The model's job shifts from "recall and predict" to "read and synthesize." That shift is the difference between an answer and a guess that sounds like an answer.
Failure Mode 1: Hedging as a Faith-Distancing Device
The hedging you notice in ChatGPT's Bible answers isn't caution. It's a side effect of RLHF training that rewards avoiding controversy over anything else. The model has no primary evidence to stand on, so instead of committing to what the text says, it retreats to "some scholars believe... while others argue..." — a move that sounds balanced but is actually just risk management for a system with nothing underneath it.
The result, in a benchmark comparing AI tools on pastoral theology questions specific to a denomination's core doctrinal statements, was stark: ChatGPT scored 0 out of 16. Not because it got facts wrong exactly — because it wouldn't commit to a position at all, even on questions where there's a clear, textually grounded answer. That's what happens when a model has no anchor beneath the prediction.
Failure Mode 2: No Sources, No Accountability
Ask ChatGPT a Bible question and you'll rarely get a real citation. You'll get a paragraph that sounds like Matthew Henry and John Gill got shoved into a blender, with no way to tell which claim came from where — or whether any claim came from anywhere at all.
That matters for a reason beyond academic tidiness. If you can't trace a claim back to a source, you can't check it, and neither can the person in your congregation who asks you about it after service. A pastor who preaches a claim they can't trace isn't just risking an embarrassing correction. They're outsourcing their accountability to a system that was never designed to hold it.
Failure Mode 3: Frozen Training Data vs. Ongoing Scholarship
This is the failure mode pastors underestimate the most. A model's training data has a cutoff. Its "knowledge" of Hittites, Ugaritic texts, or the Dead Sea Scrolls is whatever got written about those things on the internet before that cutoff — filtered again through the blending problem above.
Worse, the commentaries the model leans on most heavily in training were themselves written within the constraints of their own era. Matthew Henry (1662–1714), John Gill (1697–1771), and even Michael Heiser (1963–2023) were brilliant, but none of them had the Dead Sea Scrolls fully published, modern archaeological surveys of the Levant, or the complete Ugaritic corpus. For centuries, skeptics used the Bible's mention of the Hittites as proof of unreliability — until Hugo Winckler excavated Hattusa in 1906 and found the Hittite capital exactly where the text implied it. Older commentaries, skeptical and defensive alike, were working from incomplete data. A model trained mostly on that older material inherits its blind spots and has no mechanism to update them.
The Deuteronomy 32:8 example is even sharper. The Masoretic text reads "sons of Israel." A Dead Sea Scrolls fragment, 4QDeutj — older than the Masoretic tradition — reads "sons of God" instead, pointing toward a divine council reading of the passage that older commentators simply couldn't see, because the scroll wasn't discovered until 1947 and wasn't widely digested until decades later. An LLM answering from memory doesn't know to surface that tension. A system built to pull primary sources does.
What Grounded Architecture Changes
The fix isn't a better prompt. It's a different architecture — one that treats the Bible text, the original languages, and the historical and archaeological record as the truth layer, and treats everything else, including AI, as supplementary.
Here's the layered structure that makes the difference:
- Primary evidence, the anchor. The Bible text itself, the original Hebrew and Greek, ANE historical context, and archaeological data — the layer nothing else is allowed to override.
- Scholarly commentary, supplementary. Matthew Henry, John Gill, Michael Heiser, and others are useful, but they're inputs to be weighed against the evidence, not the evidence itself.
- AI synthesis, not generation. The model's job is to synthesize what layers 1 and 2 actually say into a coherent answer — never to generate theology from its own training-data instincts.
- Structured exploration. A real answer opens into further study rather than closing the question with a single confident paragraph.
- User agency, the final layer. The system presents evidence. The person reading it reaches their own conclusion. The theologian is still you.
The mechanical piece that makes this trustworthy in practice is citation labeling: every claim gets tagged as either verified — traced to a specific named source in the evidence set — or training-assisted — drawn from the model's training but still pointing to a real, citable work. That label sits visibly next to the claim, not buried in a tooltip. You always know which kind of answer you're looking at.
OpenLumin is the system I built around exactly this architecture — you can see it in action at OpenLumin. It's the clearest example I know of what evidence-first, retrieval-grounded design looks like when it's actually shipped, not just described. And it's the same category of system I build for ministries in my AI development work — grounded, source-anchored, and legible about where every claim comes from.
What to Ask of Any AI Tool Your Church Adopts
Before your church adopts an AI tool for Bible study or sermon prep, ask it these questions directly:
- Does it cite sources, or does it just sound confident? If you can't trace a claim to a named work, you can't verify it, and neither can your congregation.
- Are labels visible, not hidden? A tool that's honest about what's verified versus what's model-generated will show you the difference in the interface, not bury it.
- Does it treat scholars as supplementary, not final? Commentaries are valuable but era-bound. A good system weighs them against primary evidence rather than parroting them.
- Does it leave the conclusion to you? Any tool that hands you a settled theological verdict instead of the evidence behind it is doing your thinking for you. That's not a feature.
I wrote more about keeping that boundary intact — using AI as a collaborator instead of an oracle — in 13 Principles for AI-Powered Sermon Prep. The same discipline applies whether you're prepping a sermon or answering a hard question from your congregation.
The Theologian Is Still You
AI is a research assistant, not a theologian. That's not a limitation to apologize for — it's the correct division of labor. The evidence should stay in control, not the model, and not whoever trained it. Whoever controls the model controls the theology, unless the model is built to synthesize evidence and hand the conclusion back to the person doing the thinking.
If your ministry is looking at AI tools and wants something grounded in real sources instead of vibes, that's the kind of system I build. Take a look at recent work, or get in touch if you want to talk through what a grounded system would look like for your church.