IP Library Granted Patent US 12664813
Granted Patent B1
US 12664813 · App. 19/307,126 · Granted Jun 23, 2026

Method and system for transforming legacy lab notebooks into chemical intelligence and drug discovery insights using optical chemical structure recognition and natural language processing to extract knowledge from handwritten lab records

Inventors: Shobhan Shah (San Jose, CA); Aarshit Mittal (Ghaziabad, IN); Parmi Shah (Cupertino, CA)
G06V30/422G06F16/9024G06N3/08G06V30/19093G06V30/274
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12664813
App. No.
19/307,126
Granted
Jun 23, 2026
Kind
B1
Abstract

Disclosed is a computer-implemented method that includes receiving a digital image or a scanned page of a historical lab notebook that contains handwritten text and a chemical structure drawing. The method includes performing optical character recognition on the image or the page to convert handwritten text into machine-readable text data. The method includes performing optical chemical structure recognition on the image or the page to identify hand-drawn chemical structures and reaction diagrams. The method includes translating each chemical structure and reaction diagram into a standardized digital representation. The method includes analyzing the recognized text data with a natural language processing engine to extract scientific context and metadata. The metadata includes identifying a chemical entity, a reaction condition, an experimental parameter, or a result. The method includes correlating the output of the chemical structure recognition and the natural language processing by associating each chemical structure with its textual context.

Claims (52)

1 . A computer-implemented method, the method comprising:

receiving at least one of a digital image and a scanned page of a historical lab notebook containing a handwritten text and a chemical structure drawing;

performing an optical character recognition (OCR) on at least one of the digital images and the scanned pages to convert the handwritten text in the historical lab notebooks into machine-readable text data;

performing an optical chemical structure recognition (OCSR) on at least one of the digital images and the scanned pages to identify at least one of a hand-drawn chemical structure and a reaction diagram, and translating each identified chemical structure and each associated identified reaction diagram into a standardized digital representation suitable for search and computation;

analyzing recognized text data in at least one of the digital images and the scanned pages with a natural language processing (NLP) engine to extract scientific context and metadata, including identifying at least one of a chemical entity, a reaction condition, an experimental parameter, and results mentioned in the historical lab notebooks;

correlating output of the OCSR module and the NLP engine, by associating each chemical structure with its corresponding textual context from the historical lab notebooks, thereby generating a structured record of experiments that link a chemical compound to experimental details and outcomes;

constructing a structured knowledge base from a structured record, the structured knowledge base comprising an organized collection of chemical information and experimental metadata extracted from the historical lab notebooks, wherein the structured knowledge base is indexed by at least chemical structure identifiers and textual keywords;

at least one of training and configuring an artificial intelligence model using data in the structured knowledge base, such that the artificial intelligence model encodes a representation of scientific knowledge, the historical lab notebooks, and problem-solving behavior reflected in actions taken by a scientist author of each of the historical lab notebooks; and

providing an interactive user interface to the artificial intelligence model through which a user can submit at least one query and engage with the artificial intelligence model to obtain at least one of an insight, an explanation, and a prediction related to drug discovery, the artificial intelligence model responding based on information and inferred knowledge derived from the historical lab notebooks.

2 . The method of claim 1 , wherein performing the optical chemical structure recognition comprises converting each of a detected chemical structure drawing into at least one of a unique textual identifier and coding, including at least one of a SMILES string, an International Chemical Identifier (InChI), and a graph-based data structure representing the molecule's atoms and bonds.

3 . The method of claim 1 , wherein the natural language processing includes a named entity recognition to identify at least one of a chemical name, an experimental action, a reagent, the reaction condition, and a result metric in the text, and wherein identified entities are used to tag and contextualize corresponding chemical structures in the structured knowledge base.

4 . The method of claim 1 , further comprising detecting and reconstructing a chemical reaction pathway from the historical lab notebooks by identifying reaction arrows and multiple chemical structures in proximity, and creating structured records that link reactant structures to product structures, along with any associated reaction yield.

5 . The method of claim 1 , wherein associating each of the chemical structure with its textual context includes determining a role of a chemical in an experiment, the role selected from a group consisting of at least one of a reactant, a product, a catalyst, a solvent, and an intermediate, by analyzing textual cues and placement of the chemical structure in an original notebook diagram.

6 . The method of claim 5 , further comprising populating the structured knowledge base with additional metadata for each recognized chemical structure by at least one of retrieving and calculating chemical and biological properties for the chemical structure, including molecular weight, physicochemical properties, and known bioactivity data, and storing said properties in association with the chemical structure as part of an experiment record.

7 . The method of claim 6 , wherein the structured knowledge base is indexed to support substructure and similarity searching, allowing the user to query the structured knowledge base by providing the chemical structure and retrieving all notebook entries in which at least one of a matching and similar structure appears.

8 . The method of claim 1 , wherein the structured knowledge base is implemented as a graph-based ontology, in which nodes represent at least one of the chemical compounds, experiments, and observations, and edges represent relationships between them, such that the structured knowledge base captures relationships selected from: “compound A was used in experiment X,” “compound B was produced in experiment Y,” and “experiment Z had outcome result R”.

9 . The method of claim 1 , wherein training the artificial intelligence model comprises fine-tuning a domain-specific large language model on a textual content of the historical lab notebooks and a structured metadata, thereby imparting the artificial intelligence model with an understanding of historical experiments and problem-solving approaches of scientists who authored handwritten lab notebooks.

10 . The method of claim 1 , wherein the artificial intelligence model is configured to simulate scientific reasoning of at least one of a particular researcher and a group of researchers reflected in the handwritten lab notebooks, such that the artificial intelligence model, in response to a user query, to explain how a past scientist might have at least one approached a current research problem and proposed a rationale based on historical experimentation data.

11 . The method of claim 1 , further comprising a query interface that accepts natural-language questions from the user regarding at least one of experimental data and a potential new experiment, and wherein the artificial intelligence model to generate at least one of answers and suggestions by referencing the structured knowledge base, thereby enabling interactive dialogue with an agentic scientist.

12 . The method of claim 1 , wherein a system further utilizes the structured knowledge base to train at least one predictive model that analyzes aggregated historical experiment data to predict outcomes of candidate chemical reactions and to recommend optimal reaction conditions for a given synthetic goal, thereby augmenting the artificial intelligence model with prescriptive analytics capabilities.

13 . The method of claim 1 ,

wherein the method further comprises at least one of extracting and linking such experimental result data, including at least one of NMR spectra, IR spectra, melting points, and other analytical data, to corresponding experiments in the structured knowledge base, so that the artificial intelligence model to consider an empirical result when answering queries, and

wherein the historical lab notebooks include embedded data from instruments, including at least one of spectra, chromatograms, and assay results.

14 . The method of claim 1 , wherein the OCR module and the OCSR module are trained to handle handwritten and low-quality documents, including variability in handwriting styles and drawing clarity, by using deep neural network models that have been pre-trained and fine-tuned on samples of historical scientific documents, thereby improving recognition accuracy on archival notebook pages.

15 . The method of claim 1 , wherein the structured knowledge base is maintained in at least one of a secure, private cloud and on-premises environment of an organization, and further wherein the AI model is restricted to using a proprietary notebook data of the organization to train and for inference, thereby preserving confidentiality while leveraging historical data unique to that organization for drug discovery insights.

16 . The method of claim 1 , further comprising periodically updating the structured knowledge base and updating the artificial intelligence model as new historical lab notebooks become available, thereby creating a persistent, evolving digital knowledge asset that continuously learns from ongoing research and remains up-to-date.

17 . The method of claim 1 , wherein the interactive interface to provide at least one of an explanation and a citation from the historical lab notebooks alongside an answer from the artificial intelligence model, thereby ensuring transparency and allowing the user to trace a conclusion of the artificial intelligence model to underlying experimental evidence in the historical lab notebooks.

18 . The method of claim 1 , further comprising registering an identified chemical structure and associated identified reaction diagram into a private, queryable chemical knowledge base, wherein each structure is stored along with its assigned experimental role, predicted properties, and contextual metadata to enable future reuse in experimental design.

19 . The method of claim 1 , wherein the identified chemical structure and associated identified reaction diagram are directly usable within an electronic lab notebook (ELN), enabling researchers to initiate new experiments using system-registered compounds, their assigned roles, and corresponding predicted chemical and biological properties.

20 . The method of claim 1 , further comprising calculating at least one of a predicted property for each identified compound, the predicted property selected from the group consisting of: solubility, log P, pKa, ADMET profile, hERG inhibition, BBB permeability, and binding affinity to a specified protein target.

21 . The method of claim 1 , wherein the structured knowledge base supports real-time delivery of recognized compounds, reactions, and predictive insights into an ELN interface, thereby enabling a closed-loop workflow from archival extraction to experimental execution.

22 . A system for transforming unstructured lab notebook data into a structured knowledge base and an interactive artificial intelligence assistant, the system comprising:

one or more processors and memory coupled thereto;

an image processing module configured to ingest and preprocess scanned images of a historical lab notebook;

an optical character recognition module configured to recognize and output text from handwritten sections of the historical lab notebook;

an optical chemical structure recognition module configured to detect graphical depictions of at least one chemical structure and reactions on the pages and convert them into machine-readable chemical representations;

a natural language processing module configured to analyze a textual content and extract entities and relationships pertinent to chemical experiments, including a chemical name, a reaction condition, and a result;

a data storage subsystem for storing the structured knowledge base that integrates a recognized chemical structure with the extracted textual metadata, forming an indexed archive of experiments; and

an AI inference engine operatively connected to the structured knowledge base and configured to at least one respond to a user query and perform analytical tasks by utilizing information in the structured knowledge base, thereby functioning as a virtual scientific assistant derived from the unstructured lab notebook data.

23 . The system of claim 22 , wherein the data storage subsystem is deployed within at least one of a private-cloud or on-premises network isolated by a zero-trust security perimeter, thereby preserving the confidentiality of proprietary historical lab notebook data.

24 . The system of claim 22 , wherein the structured knowledge base is implemented as a graph-based ontology in which nodes represent at least one of the chemical compounds, experiments, and observations, and edges represent relationships between them.

25 . The system of claim 22 , wherein the optical character recognition module and the optical chemical structure recognition module each comprise deep-neural-network models pre-trained and fine-tuned on handwriting samples obtained from archival scientific documents to improve recognition accuracy of low-quality pages.

26 . The system of claim 22 , wherein the historical lab notebook pages are periodically ingested, the resulting experimental records are incorporated into the structured knowledge base, and the AI inference engine is refreshed, thereby enabling continuous learning from ongoing research activity.

27 . A non-transitory computer-readable medium storing a program instruction that, when executed by one or more processors, cause the processors to perform the computer-implemented method of:

detecting a hand-drawn depiction of at least one of a chemical structure and a reaction diagram on handwritten pages of the historical lab notebooks;

converting a handwritten depiction into a machine-readable chemical representation;

analyzing a textual content pertinent to a chemical name, a reaction condition, and a result surrounding the hand-drawn depiction;

extracting entities and relationships from a textual content and the hand-drawn depiction; and

transforming an unstructured lab notebook into a structured knowledge base.

28 . The method of non-transitory computer-readable medium of storing the program instructions of claim 27 , wherein the program instructions further cause the processors to calculate at least one of a predicted property for each identified compound, the predicted property selected from a group consisting of solubility, log P, pKa, ADMET profile, hERG inhibition, BBB permeability, and binding affinity to a specified protein target.

29 . The method of non-transitory computer-readable medium of storing the program instructions of claim 27 , wherein the program instructions further comprise updating instructions that, on a scheduled basis, retrain a named-entity-recognition model and refresh a graph-based ontology when additional historical notebook data is ingested.

30 . The method of non-transitory computer-readable medium of storing the program instructions of claim 27 , wherein the program instructions further cause the processors to render, via an interactive user interface, both a natural-language query entry field and a structure-based query input configured to accept SMILES, InChI, and MOL file uploads, thereby allowing a user to query the structured knowledge base by semantic or chemical-structure criteria.