DICOM de-identification software evaluation should begin with a benchmark dataset that has a known correct answer, not with a feature list. Public resources from the Medical Image De-Identification Task Group and The Cancer Imaging Archive provide synthetic DICOM files with planted identifiers, mapping files, and a validation script that checks whether each expected change occurred [2][6]. The MIDI project found that no automated method achieved a 100% success rate in fully removing protected health information or personally identifiable information [8]. That finding reframes evaluation: the question is not whether a tool claims to de-identify, but how its output performs against a documented ground truth and where human review remains necessary. A defensible evaluation therefore combines metadata checks, pixel-text inspection, UID consistency review, and a defined ownership boundary for the resulting records.
Start with a Ground-Truth Dataset, Not a Default Profile
A meaningful evaluation requires inputs where the correct de-identification actions are already known. The MIDI-B resources include a synthetic dataset with artificially generated PHI and PII inserted into DICOM headers and burned into pixel data, accompanied by an answer key and mapping files [6]. The validation script compares de-identified files against that answer key, assessing whether expected values were changed and whether UID mappings remain consistent across the dataset [2]. This structure lets an evaluator measure false positives and false negatives rather than relying on a vendor's assertion that a profile was applied.
Default configurations are a weak basis for comparison. In a study of ten free DICOM toolkits, only five could properly de-identify the defined DICOM elements, and in four cases only after careful customization [4]. The same study reported that two of the ten tools had a default-setting success rate above 90%, while four achieved 25% or less. One tool removed all required elements with its default setting; that complete-removal result is a separate measure from exceeding 90% [4]. An evaluation that runs each candidate with its out-of-the-box profile may therefore measure configuration defaults rather than the software's actual capability. The test plan should specify the exact profile, options, and element-level rules before execution.
Separate Metadata De-identification from Pixel and Overlay Risks
DICOM de-identification spans at least three distinct surfaces: structured header attributes, burned-in pixel text, and non-pixel graphics or overlays. The DICOM standard states that a de-identifier should ensure no identifying information is burned into pixel data, either because the modality does not generate it or by removing it through the Clean Pixel Data Option [7]. If non-pixel graphics or overlays contain identification, the de-identifier is required to remove them or clean them when the Clean Graphics Option is supported [7]. An evaluation that checks only header tags will miss these risks.
A documented workflow from Duke University separated the problem into metadata de-identification, secondary capture filtration, and OCR pixel de-identification [3]. In that quality improvement project, secondary capture filtration identified and removed 68,111 images, or 16%, as potentially text-bearing [3]. The authors reported zero false negatives for burned-in text across several modalities in their assessed set, but also found additional burned-in dates and times in portable DX examinations during further validation [3]. This illustrates why pixel-level checks should be modality-aware and why a single pass may not be sufficient.
A Suggested Acceptance Workflow for a Vendor Evaluation
One defensible approach is to run each candidate tool against the MIDI-B synthetic validation set and compare its output with the provided answer key and mapping files. The validation script expects three main inputs: the de-identified DICOM files, the answer key, and the mapping files [2]. After execution, review the script's report for missed header changes, burned-in text that was not obscured, and UID mapping inconsistencies. Then run a second pass on a small set of Secondary Capture or Encapsulated PDF objects to see whether the tool detects and handles them specially [1].
This workflow is a suggestion, not a published benchmark result. It does not establish legal compliance or clinical safety. It can, however, produce a documented comparison of false positives, false negatives, and referential integrity across tools. The MIDI Task Group also recommends creating additional test and evaluation datasets with known correct answers, including direct and indirect identifiers and realistic sensitive attributes [1]. Organizations with unusual modalities or proprietary private tags may need to build their own answer key rather than rely solely on public resources.
Use UID and Mapping Consistency as a Structural Check
De-identification is not only about removing names. The DICOM standard requires implementations claiming conformance as a de-identifier to protect the SOP Instance UID and all references to other SOP Instances that could be used to identify the patient [7]. The MIDI validation resources include a Patient ID Mapping File and a UID Mapping File that cross-reference identifiers changed during curation of the synthetic dataset [2]. These files enable traceability between original and de-identified datasets, allowing an evaluator to confirm that references inside sequences were updated consistently.
A practical evaluation should therefore check referential integrity, not just the presence of a new UID. If a Study Instance UID is replaced but a series-level reference still points to the old value, the output may be internally inconsistent or linkable. The validation script described in the MIDI resources systematically evaluates UID mapping consistency across the dataset [2]. This is a useful model for an internal acceptance test: after running a candidate tool, compare every UID-bearing attribute and sequence reference against the mapping file and flag any orphaned or unchanged identifier.
Treat Human Review as a Measured Component, Not a Failure
The MIDI project's final report states that in all iterations of the MIDI pipeline and with all challenge submissions, no automated method had a 100% success rate in fully removing PHI/PII [8]. The project concluded that a production setting would require a human-in-the-loop to achieve the desired threshold of 100% accuracy, while still reducing workload compared with human-only manual review [8]. This is a documented boundary for automated tools, not a defect unique to any one vendor.
An evaluation plan should therefore estimate the human review burden that remains after automation. The MIDI Task Group recommends that human quality control address structured and unstructured text, pixel data, geometric and bitmapped overlays, and compressed bitstream embedded metadata, with the percentage and type of records inspected guided by a documented risk assessment [5]. Scanned documents encountered as DICOM Secondary Capture or Encapsulated PDF objects may require OCR or manual curation and should be detected for exclusion or special handling [1]. These are measurable acceptance criteria, not optional add-ons.
Define Record Ownership and Deployment Boundaries Before Testing
Evaluation should clarify who owns the original identifiers, the mapping keys, and the de-identified output. The DICOM Basic Application Level Confidentiality Profile allows an application to create an Encrypted Attributes Data Set and copy protected attributes into the Modified Attributes Sequence, so authorized personnel can recover identity [7]. This means a tool may retain recoverable identifiers inside the output object, which has different ownership and access implications than irreversible removal. An evaluator should ask whether the tool encrypts and retains attributes, and where the decryption keys reside.
Deployment boundaries also affect what a test can establish. The MIDI project team recommended an NIH-supported standalone cloud-enabled service, but noted that this would require an Authority to Operate and documented human-in-the-loop processes [8]. A local script and a hosted service may produce different outputs for the same input depending on configuration, version, and review workflow. The evaluation should record the exact software version, profile options, and execution environment, because a result from one deployment does not transfer automatically to another.
Sources & references
- Report of the Medical Image De-Identification (MIDI) Task Group - Best Practices and Recommendations · arxiv.org
- Medical Image De-Identification Resources: Synthetic DICOM Data and Tools for Validation · arxiv.org
- A Method for Efficient De-identification of DICOM Metadata and Burned-in Pixel Text - PMC · pmc.ncbi.nlm.nih.gov
- Free DICOM de-identification tools in clinical research: functioning and safety of patient privacy - PMC · pmc.ncbi.nlm.nih.gov
- Summary of the National Cancer Institute 2023 Virtual Workshop on Medical Image De-identification—Part 1: Report of the MIDI Task Group - Best Practices and Recommendations, Tools for Conventional Approaches to De-identification, International Approaches to De-identification, and Industry Panel on Image De-identification - PMC · pmc.ncbi.nlm.nih.gov
- MIDI-B-Test-MIDI-B-Validation - The Cancer Imaging Archive (TCIA) · www.cancerimagingarchive.net
- E Attribute Confidentiality Profiles (Normative) · dicom.nema.org
- Final Report: Medical Imaging De-Identification (MIDI) Project | CRDC · datacommons.cancer.gov