How to Read LawVM Benchmark Numbers

What the benchmark measures

LawVM's Finland benchmark compares replayed point-in-time text against the Finlex updated/derived comparison surface. The public headline is one metric; structural divergence remains a separate diagnostic:

• Levenshtein text distance: character-level normalized edit distance. An August 2026 Finnish replay development run reported a 0.23% mean against its declared Finlex comparison surface.
• Structural section error: section-level divergence such as missing sections, extra sections, or content mismatches. Treat this as a separate diagnostic from text distance; this page does not publish it as a current aggregate.

The comparison surface is Finlex’s updated/derived consolidated XML. The number is a dated, project-reported agreement metric. Interpret it together with corpus scope, structural diagnostics, residual classification, and source review.

Questions requiring other evidence

Legal correctness. The benchmark measures agreement with a derived/updated comparison surface. Legal conclusions require the enacted publication and other primary sources.

Completeness. Coverage is limited to the bounded development corpus. Missing base material, comparison material, or amendment text can limit inclusion.

Amendment-family coverage. The benchmark covers statutes with section structure. The curated corpus excludes statutes made primarily of tables, schedules, or unstructured containers.

What High Similarity Means

A statute at 100% similarity means LawVM and the inspected Finlex surface produce identical text under the benchmark’s normalization. Shared defects can still exist if both systems fail the same way, for example by missing a corrigendum or reproducing a source defect.

LawVM’s corrigendum pipeline illustrates this: applying published corrections in a historical subset moved replay away from the comparison surface. Whether that makes the replay legally preferable requires source review; the score alone cannot decide it.

What Low Similarity Means

A statute at 60% similarity could mean:

• LawVM has a serious replay bug (system is wrong)
• The inspected comparison surface does not include several amendments
• The source XML is structurally corrupted, so that input alone establishes neither surface
• Finlex editors restructured the content for readability (editorial divergence)
• A combination of all of the above

The similarity number points to where to investigate. The residual taxonomy says what you find when you do.

The right way to read the numbers

The benchmark is a triage instrument. It tells you:

• Where the largest divergences are
• Which statutes deserve investigation next
• Whether a code change improves or worsens the comparison surface
• Whether a known divergence class is shrinking or growing

The real verification loop is manual residual review: investigating divergences against primary sources (Säädöskokoelma), classifying the root cause, and retaining the result in the project’s internal review record.

That is why the project uses aggregate comparison diagnostics for triage and a separate review record for dispositions. Historical project review reported 22 candidates that passed its internal threshold while many other divergences remained to be classified. The public site reports this candidate count only in aggregate. See Artifacts for the snapshot boundary.