Making 5,000 Pages of Engineering Manuals Findable: A Semantic Search Pilot for State DOTs.

About the Client

NCHRP was established in 1962 by the American Association of State Highway and Transportation Officials (AASHTO) as a national, cooperatively funded research program addressing problems common across state departments of transportation. It is administered by the Transportation Research Board, part of the National Academies of Sciences, Engineering, and Medicine, in cooperation with the Federal Highway Administration, and funded through contributions from every state DOT. WSDOT, one of NCHRP’s sponsoring state agencies, publishes 60 engineering reference manuals covering everything from hydraulics to environmental compliance that its engineers rely on daily.

Industry Context

State DOTs across the country sit on enormous technical documentation libraries built up over decades, almost universally stored as static PDFs with no electronic searchability — a findability problem WSDOT shared with transportation agencies nationwide rather than one unique to Washington State. That shared nature is precisely why NCHRP, funded cooperatively by every state DOT, sponsored this as generalizable research rather than a one-off fix: any framework and methodology developed had to be replicable by other state transportation agencies facing the identical challenge, not just usable by WSDOT alone.

Current Situation

NCHRP and WSDOT jointly selected the engineering reference manuals state engineers use to plan and execute infrastructure projects as the focus of the research. The joint project’s objectives were to define a framework for classification, search, and retrieval of transportation information, and to describe successful practices for organizing and classifying DOT content. The project also aimed to develop enterprise and federated search procedures a DOT could use to make its information more accessible, and to prepare a pilot system demonstrating the enhanced findability concepts for a state’s DOT data. Iknow and Spy Pond Partners, a Boston-based management consulting firm focused on the transportation industry, were selected to perform the work.

• An enormous, unsearchable technical library. WSDOT’s 60 engineering reference manuals existed only as static PDFs with no electronic searchability, forcing engineers to manually scroll through documents to locate critical technical guidance.
• No existing structure for classification or retrieval. The content had no taxonomy, ontology, or metadata layer; a search engine could not point a user to a precise answer without first understanding what each section of text was actually about.
• A generalizable solution was required, not a one-off fix. Because the research was funded cooperatively by every state DOT through NCHRP, the pilot needed to demonstrate a replicable framework other states could adopt, not just a bespoke tool for WSDOT alone.
• Highly technical, domain-specific content. Manuals covering hydraulics, stormwater management, environmental compliance, and roadside design used precise, specialized engineering vocabulary that a generic search engine or off-the-shelf classification tool would not reliably understand.

• Define a framework for classification, search, and retrieval of transportation information.
• Describe successful practices for organizing and classifying DOT content.
• Develop enterprise search and federated search procedures a DOT could use to make its transportation information more easily accessible for users.
• Prepare a pilot system to demonstrate the enhanced findability concepts for a state’s DOT data.

How Iknow Structured the Work

Iknow applied a four-step process — Prepare, Analyze, Describe, and Categorize — to a defined corpus of eight WSDOT manuals containing stormwater-related content, totaling more than 5,000 pages of previously unsearchable text.

Key Activities & Decisions

• Prepare. Iknow deconstructed the eight manuals into sections and subsections — smaller text or HTML files each covering a single topic — so the search engine could point users to precisely the right place in a technical manual. Iknow then removed three types of “noise” (stop words, boilerplate text, and non-value-adding phrases like “Table of Contents”) before analysis.
• Analyze. Iknow applied text mining to identify and count the keywords and phrases needed for semantic autocategorization models and a project thesaurus, and cluster analysis to group textual data into conceptually meaningful classes that could be arranged hierarchically for navigation.
• Describe. Based on the text mining and cluster analysis results, Iknow selected three independent facets for the pilot: Asset (e.g., Culvert), the Master Deliverables List (a hierarchy of processes, work groups, milestones, and deliverables), and Best Management Practices for stormwater management and treatment.
• Categorize. Iknow built an ontology for each facet: a Culvert Ontology drawn from the WSDOT Hydraulics Manual, FHWA Hydraulic Design Series No. 5, and WSDOT’s Engineering Publications Thesaurus, with four branches covering shapes, parts, materials, and end treatments; a Master Deliverables List Ontology built directly in the open-source Protégé editor with “kind of” and “equivalent” term relationships; and a Stormwater BMP Ontology drawn from WSDOT’s Temporary Erosion and Sediment Control Manual, Highway Runoff Manual, and the Washington State Department of Ecology’s Stormwater Manual for Western Washington.
• Technical Implementation. Iknow assembled a best-of-breed architecture combining Drupal (using the Drupal Books module to ingest and present the manuals while preserving their original structure), Apache Solr for search, and MySQL for content storage. The architecture was layered with Protégé for ontology authoring, Apache Jena to match content against the ontology and store the results as triples, Google NLP for lemmatization and entity extraction, and Taggr — an Iknow-proprietary application built specifically to apply Jena’s identified tags to the correct section of content within Drupal. The resulting pilot delivered faceted filtering by manual, asset, and master deliverable; relevancy-ranked results with highlighted text snippets; and full-content pages that preserved each manual’s original table-of-contents navigation.

Stakeholders & Collaboration

Iknow worked as subcontractor to prime contractor Spy Pond Partners, delivering the pilot system workstream within the broader NCHRP/WSDOT research project, under the oversight of NCHRP and the Transportation Research Board.

Teaching a Search Engine to Understand Highly Technical Engineering Language

Generic search and classification tools are not built to reliably understand precise, domain-specific engineering vocabulary spanning hydraulics, stormwater management, and roadside design. Iknow addressed this by building purpose-specific ontologies grounded directly in WSDOT’s own authoritative source manuals and FHWA and state stormwater standards, rather than a generic thesaurus, and by combining rules-based semantic classification with machine learning-based “learn by example” classification so the system could apply expert-defined structure while continuing to improve through use.

Assembling a Reliable Pipeline From Six Independent Tools

The technical solution depended on six separate open-source and proprietary tools — Protégé, Apache Jena, Google NLP, Drupal, Apache Solr, and MySQL — each of which needed to hand off cleanly to the next without breaking the chain from raw text to searchable, tagged content. Iknow addressed this by designing and documenting an explicit end-to-end process schematic connecting every tool, and by building Taggr specifically to solve the one missing link in that chain: applying Jena’s identified metadata tags back to the precise correct section of content inside Drupal.

Operational Outcomes

• Processed eight WSDOT engineering manuals totaling more than 5,000 pages of previously unsearchable PDF content.
• Built three independent taxonomy facets — Asset, Master Deliverables List, and Best Management Practices — and three corresponding ontologies grounded in WSDOT’s and FHWA’s own technical standards.
• Integrated six distinct open-source and proprietary software products, including Iknow’s own Taggr application, into a single working pilot pipeline.

Strategic and Organizational Outcomes

The pilot system confirmed that findability could be meaningfully improved by analyzing the “aboutness” of a content corpus, applying precise descriptive metadata, and automating classification through a combination of semantic rules and machine learning. It delivered a faceted, browsable, relevancy-ranked search experience that preserved each manual’s original organizational logic while making its content discoverable for the first time — and, as a cooperatively funded NCHRP research project, the resulting framework was designed to be generalizable and available to other state DOTs facing the same documentation challenge. The pilot project went on to guide WSDOT’s future efforts to improve the findability and discoverability of its technical content.

Timeline to Impact

Iknow delivered a fully functional, demonstrable pilot application within the five-month engagement — a fast turnaround for a project spanning corpus deconstruction, ontology design, and the integration of six separate technologies into one working system.

Core Skills

• Text Analytics & Content Classification
• Taxonomy & Ontology Design
• Enterprise Search Implementation
• Custom Software Development

Methods & Frameworks

• Text mining and cluster analysis
• Rules-based semantic classification combined with machine learning “learn by example” classification
• Faceted search and browse design
• Corpus deconstruction and content chunking methodology

Technologies & Tools

• Protégé (ontology authoring)
• Apache Jena (ontology matching and triple storage)
• Google NLP (lemmatization and entity extraction)
• Drupal, Apache Solr, and MySQL (content management, search, and storage)
• Taggr (Iknow-proprietary auto-tagging application)