Machine Learning vs. Rules: A Head-to-Head Classification Test for Utah DOT’s Document Archive.

About the Client

NCHRP was established in 1962 by the American Association of State Highway and Transportation Officials as a national, cooperatively funded research program administered by the Transportation Research Board, part of the National Academies of Sciences, Engineering, and Medicine, in cooperation with the Federal Highway Administration. UDOT, the state agency responsible for planning, building, and maintaining Utah’s highway system, engaged with NCHRP’s Project 20-97 specifically because it was in the process of implementing Knowvation and wanted a data-driven basis for classifying its documents within the new platform.

Industry Context

This engagement was the third Iknow delivered under NCHRP’s broader Project 20-97, “Improving Findability and Relevance of Transportation Information,” following an earlier pilot and a full-scale ontology project for the Washington State Department of Transportation. Together, the three projects demonstrate how a single cooperative research program can generate reusable findability methodology tested across multiple state DOTs using different technical approaches — rules-based ontology classification for engineering manuals in Washington, and a direct empirical comparison of machine learning against rules-based classification for UDOT’s agreement archive. Precision and recall are the standard, industry-accepted metrics for evaluating classification and information-retrieval quality: precision measures what fraction of retrieved results are actually correct, recall measures what fraction of all truly relevant documents were successfully found, and industry-standard scores typically start around 60% and are improved iteratively, with 80% or higher generally considered a success.

Current Situation

The project focused on searching and finding old plans, agreements, and background data required to inform construction planning for new transportation infrastructure projects on the same or abutting sites, spanning five real-world use cases: paving project scoping, corridor studies, environmental analysis, public records requests, and research agreements. The source material documented a concrete example of the stakes involved — UDOT staff recalled a prior agreement with a railroad that could affect a planned bridge removal, but the railroad itself could not locate the agreement, and the project could not proceed without it.

• A massive, largely unclassified document corpus with a needle-in-a-haystack problem. The corpus totaled 272,000 files and 727 gigabytes, and the target agreement types — Utility, Maintenance, Cooperative, Betterment, and Drainage — represented only a small percentage of the total, making them genuinely hard to find even when critical to a project.
• Five distinct, high-stakes use cases with real consequences for missing documents. Paving project scoping, corridor studies, environmental analysis, public records requests, and research agreements all depended on finding the right prior agreement or report, as UDOT’s own stalled bridge-removal example demonstrated.
• No empirical basis for choosing between two viable classification approaches. Machine learning and rule-based classification both offered plausible paths to autoclassifying the corpus, but UDOT had no rigorous, data-driven comparison of how each would perform on its specific document types before committing to one.
• Classification alone was not enough; specific data needed to be extracted from within documents. Knowing a file was an agreement did not answer the question at hand; UDOT needed the project number, agreement parties, tax ID, and agreement date pulled out to make tagged results genuinely useful.

• Develop and apply both rule-based and machine learning methods for assigning agreement types to files, and compare the results.
• Develop entity extraction scripts to pull specific structured data fields from agreement documents, including project number, agreement party, tax ID, and agreement date.
• Create an output file with assigned tags and file references to augment Knowvation’s existing metadata.
• Produce a rigorous, precision- and recall-based comparison of the two classification approaches to guide future DOT autoclassification efforts.

How Iknow Structured the Work

Iknow developed and tested machine learning and rule-based classification approaches in parallel against the same corpus and the same five target agreement types, evaluating both using precision and recall so UDOT could see exactly how each method performed rather than relying on subjective impressions.

Key Activities & Decisions

• Machine Learning Approach. Iknow used the open-source Python 3.6 programming language, installed via the Anaconda 3 distribution, along with NLTK for text processing — tokenization, stemming, tagging, parsing, and semantic reasoning — and Scikit-learn, a machine learning library featuring classification, regression, and clustering algorithms interoperating with NumPy and SciPy. Iknow trained machine learning classifiers to identify each agreement type from labeled training examples.
• Rule-Based Classification. Iknow wrote custom logical rules for each of the five agreement types, using both direct IF-THEN category assignment and weighted, scored rules that summed term-presence evidence across defined document sections to determine a document’s category.
• Entity Extraction. Using Python’s NLTK expression-matching modules, Iknow built extraction scripts that pulled project number, agreement party, tax ID (for drainage agreements), and agreement date directly out of documents identified as agreements.
• Evaluation and Tuning. Iknow measured both approaches using precision and recall, iteratively tuning the machine learning training sets and the rule sets to the data. Adding “negative evidence” — examples of non-agreements — improved both precision and recall for both methods.
• Output Integration. Iknow created an output file containing assigned tags and corresponding file references, designed to augment the metadata already present within Knowvation.

Stakeholders & Collaboration

Iknow worked as subcontractor to prime contractor Spy Pond Partners under NCHRP’s oversight, delivering results intended for direct use by UDOT and for broader application across other state DOTs facing similar large-corpus classification challenges.

Finding Rare Content Types Reliably Within a Massive Corpus

The target agreement types represented a small fraction of a 272,000-file, 727-gigabyte corpus, creating real risk that either classification method alone could miss critical documents or bury them in false positives. Iknow addressed this by running two independent, purpose-built classification methodologies in parallel against the same corpus and target types, rather than betting on a single untested approach, and by rigorously measuring both with precision and recall so UDOT could see exactly how each performed.

Interpreting Mixed Results in a Useful, Actionable Way

Some results, like a notably low recall score for utility agreements, risked being read as method failure rather than understood correctly. Iknow addressed this by explicitly diagnosing the cause behind anomalous results — in that case, a small sample size rather than a flaw in the method. Iknow then provided concrete, method-specific paths to improvement for both approaches: more positive and negative training examples for machine learning, and false-positive-driven rule refinement for rule-based classification, turning a benchmarking exercise into an actionable improvement roadmap.

Operational Outcomes

• Analyzed a corpus of 272,000 files totaling 727 gigabytes to locate five target agreement types making up a small fraction of the total content.
• Directly compared both approaches on Maintenance Agreements: machine learning achieved 93% precision and 86% recall, while rule-based classification achieved 78% precision and 95% recall — showing a clear precision-versus-recall tradeoff between the two methods.
• Applied rule-based classification to four additional agreement types, achieving 95% precision on Utility Agreements, 82% precision and 79% recall on Betterment Agreements, 100% precision and 75% recall on Drainage Agreements, and 93% precision and 71% recall on Cooperative Agreements.

Strategic and Organizational Outcomes

Most results met or approached the industry-standard 80% success threshold for precision and recall, and Iknow’s entity extraction scripts successfully pulled structured data — project numbers, agreement parties, tax IDs, and agreement dates — directly from identified agreements, producing an output file that augmented Knowvation’s existing metadata. Because the methodology and findings were developed under NCHRP’s cooperative research program, they were explicitly designed to be applicable and replicable across other state DOTs facing similar large-corpus classification challenges, not just usable by UDOT alone.

Timeline to Impact

Iknow’s assignment lasted 18 months.

Core Skills

• Machine Learning-Based Autoclassification
• Rules-Based Classification Engineering
• Entity Extraction & Structured Data Capture
• Classification Accuracy Benchmarking

Methods & Frameworks

• Precision- and recall-based classification evaluation
• Parallel machine learning vs. rules-based methodology comparison
• Training-set and ruleset iterative tuning
• Negative-evidence-informed classification refinement

Technologies & Tools

• Python 3.6 (via Anaconda 3), NLTK, and Scikit-learn
• Knowvation (PTFS information indexing and discovery platform)