AI quality management document review workflows combine language models with deterministic controls to extract content, check it against governing documents, and surface findings for human review. The documented systems differ in scope: one reviews executed pharmaceutical batch records against master records and specifications, another assists FDA conformance reviewers with establishment inspection reports, and a third applies rule-based and LLM checks to medical records. None of the cited systems approves, rejects, or releases records. People decide on every finding, and the quality unit retains disposition authority. The technical decision points center on how review rules are versioned, how references are matched, how model outputs are bounded, and how performance is measured against human-annotated or adjudicated findings.
The Named Workflow: CDP for Executed Batch Records
Enveda describes CDP as a language-model system for quality review of executed pharmaceutical batch records [1]. The system reads each document, checks it against governing documents such as the master batch record and specification, and posts comments and an advisory outcome beside the file [7]. People review and disposition every comment, and the quality unit decides every batch. CDP does not approve, reject, release, or close any record [7]. The supplementary notes state that CDP has not been validated as a GMP computerized system and that the notes are not a validation package [7].
The review applies written rules rather than the AI model's general judgement so that the same entry is treated the same way on every record [1]. The rules for each document type are held in a versioned prompt in the ECM and are agreed with the CMC functions that own those documents [1]. Configuration items include the review prompts and rules, the reference rule set, feature switches, and the choice of AI model for each task, all held as versioned files in the ECM rather than written into the code [1]. A change of AI model requires quality sign-off and an entry in the installation-qualification register before production application [1].
Document Extraction and Reference Matching
Batch records are difficult documents to read. Most are scanned images with no text layer, filled in by hand and often bilingual, with corrections struck through and attached tables and instrument charts [1]. These records can be hundreds to thousands of pages, which is more than an AI model can carry in one request [1]. The content extractor sends the whole record to the AI model in batches of consecutive pages and verifies that no page is skipped [1]. This batching approach addresses a practical constraint that any quality document review workflow must handle when source files exceed model context limits.
A record is reviewed against the documents that govern it, such as the master batch record, the specification, and the methods it cites [7]. The supplementary notes describe a reference-finding process with a preferred tie-break field, an ordered list of looser fallback queries, whether to look up each item in a list field separately, and the dimensions a candidate must match, such as dosage form [7]. The system also keeps an AI-call log with one record for every AI model call in review, extraction, and the registries, including refused calls [7]. This logging supports auditability of model interactions.
A Suggested Evaluation Example
A quality team evaluating an AI review workflow could select a small set of executed records with known findings, run the system in capture-only or dry-run mode, and compare system comments against the known findings. The Enveda supplementary notes describe capture-only mode and dry-run mode as ways to run without writing review results to the ECM, and evaluation against documents with known findings [7]. The team would then adjudicate each comment as actionable, non-actionable, or a misread, following the approach described in the CDP study [1]. This suggestion is an editorial example, not a tested protocol from the cited sources.
FDA Conformance Review Pilots
FDA staff and Booz Allen Hamilton describe an AI-assisted tool for regulatory document conformance review of establishment inspection reports [2][3]. The pilot codified a subset of Investigation Operations Manual criteria for EIRs, and the tool extracts relevant text data from PDF EIRs for expedited conformance decisions by human reviewers [2]. After review is complete, the tool automatically compiles reviewer comments into the desired feedback form [2]. The workflow preprocesses sections to extract information up front, reducing reading for discovery [2][3].
The tool was evaluated against a hand-annotated corpus of 100 EIRs [2][3]. The cited tables report semantic search F1 scores ranging from 0.94 to 0.97, while rule-based NLP criteria generally scored highest [2][3]. The 0.90 F1 value in the tables corresponds to a dependency-parsing criterion, not semantic search. At the time of the cited material, only 19 of the 200+ IOM and Compliance Program criteria had been coded and evaluated [3]. The pilot was hosted on a CPU AWS node with access granted by reviewer team leadership on an individual basis [2][3].
Medical Record Quality Control System
A separate study describes the Medical Record Quality Control System (MRQCS), a four-tier architecture for medical record quality control [4]. The system includes a notification workflow that displays the number of detected defects without disrupting normal EHR operations [4]. After clinicians revise and resave documents, QC outputs are refreshed to form a closed-loop workflow covering documentation, quality review, and revision [4]. A supplementary LLM-based review function identifies latent defects missed by rule-based checks [4].
The study retrospectively evaluated associations with defect detection, medical record quality, and interreviewer consistency using seven-year longitudinal final-stage QC records from 2019 to 2025 [4]. The mean number of defects per sampled case increased after January 2021 and declined after July 2023 [4]. Reviewer consistency data show a defect rate coefficient of variation of 115.7% in the manual period and an intraclass correlation coefficient of 0.782 [4]. The datasets are available from the corresponding author upon reasonable request [4].
Configuration, Oversight, and Evidence Limits
The Enveda study reports that no CDP comment cited content that was not in the record [1]. The adjudicator found no comment in either run that referred to content absent from the record, though this measure does not capture misreads that produce no comment [1]. Comments that read an existing entry wrongly were counted separately as misreads, and each run had one [1]. A run using Claude Opus 5.5 provided six actionable comments that no human reviewer raised, four of which were not included in previous CDP runs, but the review time for that run was not recorded [1].
AODocs, a document management vendor, describes AI quality management use cases as assistance tasks [6]. Quality owners still decide whether a finding is valid, whether an action addresses the cause, and whether a controlled document can change [6]. The vendor recommends keeping missing-item flags separate from a decision about whether evidence satisfies a requirement [6]. It also advises recording the model or configuration used where available, the source scope, and the output, and re-evaluating the workflow when those elements or business requirements change [6].
Sources & references
- CDP: a language-model system for quality review of executed pharmaceutical batch records - Enveda · enveda.com
- AI Assisted Document Conformance Tool · www.fda.gov
- AI Assisted Document Conformance Tool · www.fda.gov
- Real-Time Medical Record Quality Control System (MRQCS): Development and Retrospective Longitudinal Observational Study - PMC · pmc.ncbi.nlm.nih.gov
- https://ieeexplore.ieee.org/document/11365130/ · ieeexplore.ieee.org
- AI Quality Management Use Cases for Document Workflows | AODocs · www.aodocs.com
- Supplementary Information · raw.githubusercontent.com
- Computer Software Assurance for Production and Quality Management System Software | FDA · www.fda.gov