truthfulqa

Production-grade LLM Evaluation & Benchmarking Framework - GPT-4, Claude, Gemini, Mistral. Accuracy, latency, cost, hallucination, reasoning metrics.

Code for NAACL paper When Quantization Affects Confidence of Large Language Models?

Enterprise-grade LLM evaluation framework | Multi-model benchmarking, honest dashboards, system profiling | Academic metrics: MMLU, TruthfulQA, HellaSwag | Zero fake data | PyPI: llm-benchmark-toolkit | Blog: https://dev.to/nahuelgiudizi/building-an-honest-llm-evaluation-framework-from-fake-metrics-to-real-benchmarks-2b90

LLM hallucination evaluation pipeline: two Claude models scored on TruthfulQA (817 questions, 38 categories) via AWS Bedrock and DeepEval LLM-as-judge, with per-category failure analysis and model comparison

A hallucination detection pipeline for Large Language Models (LLMs).

A deterministic honesty layer for LLM agents + a reproducible TruthfulQA eval that measures calibration honestly — reporting the wins and the costs.

Does inter-agent LLM disagreement signal hallucination? An 817-question TruthfulQA study with judge-based evaluation and a fair single-model baseline.

Official code for "From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems" (IWSDS 2026)

Premise-direction NLI consensus for response selection in LLMs

A hybrid 4-stage pipeline for detecting and mitigating hallucinations in Large Language Models using RAG retrieval, self-consistency sampling, uncertainty estimation, and NLI-based verification. Evaluated on TruthfulQA.

Adaptive, probe-controlled activation steering that cuts LLM hallucination rate by ~10pp (64.6%→58.5%) on TruthfulQA — steers only when a real-time risk probe flags danger, unlike fixed-strength ITI/CAA/TSV baselines reproduced here for comparison.

Evaluation of Llama-3.1-8B Base vs Instruct on TruthfulQA using few-shot prompting and automatic judge models

Evaluating how semantically equivalent prompt variations affect LLM predictions and uncertainty estimates.

Does instruction tuning make language models more sycophantic? A paired causal study across Qwen, Llama, and Gemma on TruthfulQA, showing the effect is family-dependent in both magnitude and direction. 7,200 evaluations, 12 ATE estimates with paired t-tests and bootstrap CIs.

A tool to evaluate and compare local LLMs running on Ollama or LM Studio under identical conditions using deepeval's public benchmarks (MMLU, TruthfulQA, GSM8K).

PT-GAT Transformer Diagnostics: task-relative hallucination diagnosis with adequacy triggers, evidence conditioning, and anti-collapse baselines.

Multilingual hallucination evaluation framework for Large Language Models across Indian languages using TruthfulQA, NLLB-200, and mechanistic interpretability.

Multi-agent framework for hallucination detection and correction in LLM outputs using retrieval-grounded verification. MSc AI/ML dissertation (LJMU).

CAP6640-Spring2026: Benchmarks GPT-3.5, GPT-4, Claude Haiku, and Gemini on GSM8k and TruthfulQA, measuring accuracy, self-consistency, and confidence calibration.