fleiss-kappa

The tool creates a visualization of the video annotation matrix. It also converts a labeled video matrix into a Fleiss Matrix. Finally, it calculates the Overall Fleiss Kappa Score, Percent Overall Agreement among raters above chance, Confidence Interval of Kappa & Significance Test

Multi-label inter rater agreement using fleiss kappa, krippendorff's alpha and the MASI similarity measure for set simmilarity. Written in R Quarto.

💬 Advanced NLP with Spacy Course

𝗟𝗮𝗯𝗥𝗜 𝗦𝗵𝗶𝗻𝘆 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻: Tool for estimating reference intervals using the indirect sampling method.

Cross-model LLM-as-judge eval harness: validate AI judges with Fleiss' kappa / Krippendorff's alpha, not accuracy. Ships a real 7-model panel (Claude, GPT, Gemini, Grok, Qwen, DeepSeek, GLM) you can replay in ~30s, no API key. MIT.

Raters inter-rater agreement

Evaluation and agreement scripts for the DISCOSUMO project. Each evaluation script takes both manual annotations as automatic summarization output. The formatting of these files is highly project-specific. However, the evaluation functions for precision, recall, ROUGE, Jaccard, Cohen's kappa and Fleiss' kappa may be applicable to other domains too.

The program implements the calculus of the Fleiss' Kappa in the both the fixed and margin-free version. The data used are a collection of possible results of an analysis of 6 annotators on 100 texts.

A machine learning-based career prediction system that analyzes personality traits (OCEAN model) from CSV data or video interviews to recommend multiple career roles. Supports Random Forest, SVM, KMeans, and facial emotion analysis with DeepFace.

Agreement and consensus benchmarking for LLM classification

Using information retrieval to help an emerging local artist boost its streams

Statistical validation of labeling consistency across three independent raters for a handwritten digit classification dataset.

Data and codes for analysing an industry model Pacific oyster for sustainable aquaculture

Validate LLM-as-judge reliability with inter-rater agreement, not accuracy. Reproduces a TypeScript reference implementation on a real 7-model panel.

Production-grade pipeline for validating annotation consistency and evaluating LLM output quality using agreement metrics, schema validation, LLM-as-judge scoring, SQLite logging, and a Streamlit dashboard.