An accuracy number is meaningful only with a documented test design
CER and WER compare OCR with a trusted reference, but the result is meaningful only when sampling, normalization, alignment, and ground-truth rules are documented.
The final report should make its ground truth, normalization, alignment, error unit, sample, and exclusions visible so another reviewer can understand what the accuracy number measures.
Build a review sample with OCR Quality Checker. Use the next control in Sample-Based OCR Quality Assurance for Large Collections and OCR Confidence Scores: What They Can and Cannot Tell You.
Accuracy-reporting controls
- Write ground-truth rules
- Version the reference set
- Document normalization
- Report CER and WER by source class
- Add field and layout measures
Define ground truth, normalization, alignment, and error units
Create independent ground truth
Transcribe representative samples under a written policy for spelling, punctuation, line breaks, unreadable text, and layout. Double-check the reference because ground-truth mistakes are counted as OCR mistakes.
Define the metric
Character error rate uses insertions, deletions, and substitutions relative to reference characters. Word error rate applies the same idea to word tokens. State how spaces, case, Unicode normalization, and punctuation are treated.
Report strata and distributions
Break results down by document type, language, capture method, layout, and difficulty. Averages can conceal a small class with unacceptable performance or one catastrophic page.
Add task-specific measures
For invoices, evaluate exact totals and dates; for archives, evaluate names and search recall; for accessibility, evaluate reading order and structure. CER alone does not measure these properties.
Make the measurement reproducible
Create reviewed ground truth for a documented sample. Decide in advance how to handle case, whitespace, punctuation, hyphenation, Unicode normalization, and reading order. Align OCR output with the reference before counting substitutions, deletions, and insertions.
CER is sensitive to character differences; WER reflects word boundaries and can magnify spacing or tokenization choices. Report the source classes, sample size, normalization rules, metric formula, and uncertainty or limitations. Keep a separate list of consequential field errors because one aggregate rate may hide a wrong name or total.
- Freeze normalization rules
- Retain ground truth and alignment
- Report consequential errors beside aggregate rates
Metric selection
| Metric | Measures | Misses |
|---|---|---|
| CER | Character edits | Field consequence and layout |
| WER | Word-token edits | Internal character severity |
| Exact field match | Whole value correctness | Errors outside selected fields |
| Reading-order review | Sequence usability | Character correctness |
One rate cannot describe every error
CER and WER change with ground truth, normalization, alignment, tokenization, and sample design. They may also hide a small number of high-consequence field errors.
What makes a measurement reproducible
The sources define metrics and evaluation principles. Publish the reference text, rules, sample, exclusions, and calculation method so another reviewer can interpret the number.