Assessing the Concordance Between LLM-Based ICD-10 Coding and a Drug Database for Antihypertensive Drug Contraindications: A Weighted Inter-Rater Agreement Study

Stud Health Technol Inform. 2026 Jun 29;338:539-543. doi: 10.3233/SHTI260902.

ABSTRACT

BACKGROUND: Manual ICD-10 coding of drug contraindications is resource-intensive and difficult to scale. Large language models (LLMs) have shown early promise for clinical information extraction, but their reliability for structured pharmacological coding relative to validated expert databases remains unknown.

OBJECTIVE: To assess the concordance between LLM-based automated ICD-10 coding and Thésorimed drug database1 for antihypertensive drug contraindications.

METHODS: Contraindications were extracted from Summary of Product Characteristics free-text documents for 301 antihypertensive drugs using a retrieval-augmented generation pipeline and aligned against Thésorimed ICD-10 codes. Agreement was assessed using a binary kappa (κ1, entity presence) and a hierarchical weighted kappa (κ2, ICD-10 code concordance) on 5,074 entities.

RESULTS: Entity-level agreement was slight (κ1 = -0.30), reflecting structural differences in source coverage. Code-level agreement among matched pairs was substantial (κ2 = 0.70, Po = 0.871), with 76.9% identical codes.

CONCLUSION: When both sources identify the same contraindication, LLM-based coding shows substantial concordance with expert-curated ICD-10 assignments. However, low entity-level agreement highlights important differences in coverage and representation between sources.

PMID:42394064 | DOI:10.3233/SHTI260902