Semantic multimodal search of engineering documents
In example embodiments, a semantic multimodal search function is provided in an engineering application for searching engineering documents. In an indexing phase, the application indexes a library of engineering documents to build an embedding database. For each of the engineering documents, a region detector extracts regions that each correspond to a different mode of technical or engineering data, and a set of ML models is used to generate embeddings that represent the semantic significance of technical or engineering data in each of the regions. In a query phase, the application receives search input and uses the region detector to extract regions that each correspond to a different mode of input. The application uses the ML models to generate embeddings that represent the semantic significance of input in each of the regions, and then performs a vector search between these embeddings and those in the embedding database.
1 . A method for searching for engineering documents, comprising:
receiving, by an engineering application executing on one or more computing devices, search input from a user;
extracting from the search input, by a region detector of the engineering application, a plurality of regions that each correspond to a different mode of input of a plurality of modes of input, wherein the plurality of regions of the search input includes at least one region that corresponds to schematic input or computer aided design (CAD) model input, and at least one other region that corresponds to text input or image input;
generating one or more query embeddings that represent a semantic significance of input in each of the plurality of regions using a plurality of machine learning (ML) models of the engineering application, wherein the plurality of ML models includes a ML model adapted to produce first query embeddings that represent the semantic significance of the schematic input or CAD model input and a ML model adapted to produce second query embeddings that represent the semantic significance of text input or image;
comparing, by the engineering application, the one or more query embeddings and a plurality of document embeddings of an embedding database that each correspond to a respective engineering document maintained in a library of engineering documents, and selecting one or more engineering documents having closest document embeddings; and
displaying search results based on the selected one or more engineering documents.
2 . The method of claim 1 , wherein the first and second query embeddings are temporary query embeddings, and the generating the one or more query embedding comprises:
generating for each of the plurality of regions, by a respective ML model adapted to the mode of input in the respective region, a temporary query embedding that represents the semantic significance of input in the respective region to produce a set of temporary query embeddings;
aggregating, by an aggregation ML model of the engineering application, two or more of the set of temporary query embeddings to produce a main query embedding; and
using the main query embedding in the one or more query embeddings.
3 . The method of claim 1 , wherein the generating the one or more query embeddings comprises:
determining for a region of the plurality of regions that contains a mode of input other than text input, a text description describing the semantic significance of input in the respective region;
generating, by a ML model adapted to text input, a query embedding based on the text description; and
including the query embedding in the one or more query embeddings.
4 . The method of claim 1 , wherein the comparing the one or more query embeddings and the plurality of document embeddings comprises:
determining a distance between the one or more query embeddings and the plurality of document embeddings; and
selecting the one or more engineering documents whose document embeddings have a smallest distance.
5 . The method of claim 1 , further comprising:
receiving, by the engineering application, filter input from the user;
limiting the plurality of document embeddings of an embedding database subject to the comparison with the one or more query embeddings based on a comparison of the filter input and metadata extracted from engineering documents.
6 . The method of claim 1 , further comprising:
receiving, by the engineering software, query refinement text from the user;
generating, by an additional ML model of the engineering application, a score for each of the search results based on the query refinement text, the search input, and the search results themselves; and
refining or reordering the search results based on the scores.
7 . The method of claim 1 , further comprising:
indexing the library of engineering documents to build the embedding database by:
for each of a plurality of engineering documents in the library of engineering documents,
extracting from the engineering document, by the region detector, one or more regions that each correspond to a different mode of technical or engineering data of a plurality of modes of technical or engineering data, wherein the plurality of modes of technical or engineering data include text data, image data, schematic data and CAD model data;
generating, by the one or more ML models, one or more document embeddings that represent a semantic significance of technical or engineering data in each of the one or more regions; and
storing the one or more document embeddings in the embedding database.
8 . The method of claim 7 , wherein the generating the one or more document embeddings comprises:
generating for each of the one or more regions, by a respective ML model adapted to the mode of technical or engineering data in the respective region, an embedding that represents the semantic significance of technical or engineering data in the respective region to produce a set of temporary document embeddings;
aggregating, by the aggregation ML model, two or more of the set of temporary document embeddings to produce a main document embedding; and
including the main embedding in the embedding database.
9 . The method of claim 7 , wherein the generating the one or more document embeddings comprises:
determining for a region of the one or more regions that contains a mode of technical or engineering data other than text data, a text description describing the semantic significance of technical or engineering data of the respective region;
generating, by a ML model adapted to text data, a document embedding based on the text description; and
including the document embedding in the embedding database.
10 . The method of claim 7 , wherein the indexing the library of engineering documents comprises:
for each of a plurality of engineering documents in the library of engineering documents,
extracting metadata from the engineering document; and
storing the extracted metadata in connection with the one or more document embeddings.
11 . The method of claim 10 , wherein the metadata includes augmented metadata that represents classifications or tags for symbols or elements within at least one of the one or more regions.
12 . A non-transitory computing device readable medium having instructions stored thereon, the instructions when executed by one or more computing devices operable to:
index a library of engineering documents to build an embedding database by:
for each of a plurality of engineering documents in the library of engineering documents,
extracting from the engineering document one or more regions that each correspond to a different mode of technical or engineering data,
generating one or more document embeddings that represent a semantic significance of technical or engineering data in each of the one or more regions, and
storing the one or more document embeddings in the embedding database;
query the library of engineering documents by:
receiving search input,
extracting from the search input a plurality of regions that each correspond to a different mode of input of a plurality of modes of input, wherein the plurality of regions of the search input includes at least one region that corresponds to schematic input or computer aided design (CAD) model input, and at least one other region that corresponds to input other than schematic input or CAD model input;
generating one or more query embeddings that represent a semantic significance of input in each of the plurality of regions, wherein the generating produces first query embeddings that represent the semantic significance of the schematic input or CAD model input and second query embeddings that represent the semantic significance of text input or image; and
comparing the one or more query embeddings and the one or more document embeddings of the embedding database, and selecting one or more engineering documents based on the comparison; and
display search results based on the selected one or more engineering documents.
13 . The non-transitory electronic-device readable medium of claim 12 , wherein the instructions to index the library of engineering documents comprise instructions that when executed are operable to:
generate for each of the one or more regions a temporary document embedding that represents the semantic significance of technical or engineering data in the respective region to produce a set of temporary document embeddings;
aggregate two or more of the set of temporary document embeddings to produce a main document embedding; and
include the main embedding in the embedding database.
14 . The non-transitory electronic-device readable medium of claim 12 , wherein the instructions to index the library of engineering documents comprise instructions that when executed are operable to:
determine for a region of the one or more regions that contains a mode of technical or engineering data other than text data, a text description describing the semantic significance of technical or engineering data of the respective region;
generate a document embedding based on the text description; and
include the document embedding in the embedding database.
15 . The non-transitory electronic-device readable medium of claim 12 , wherein the instructions to query the library of engineering documents comprise instructions operable to:
determine a distance between the one or more query embeddings and the plurality of document embeddings; and
select the one or more engineering documents whose document embeddings have a smallest distance.
16 . The non-transitory electronic-device readable medium of claim 12 , wherein the input other than schematic input or CAD model input includes text input or image input, and the plurality of modes of technical or engineering data include text data, image data, schematic data and CAD model data.
17 . The non-transitory electronic-device readable medium of claim 12 , wherein the instructions when executed are further operable to:
receive filter input; and
limit the plurality of document embeddings of the embedding database subject to the comparison with the one or more query embeddings based on a comparison of the filter input and metadata extracted from engineering documents.
18 . The non-transitory electronic-device readable medium of claim 12 , wherein the instructions when executed are further operable to:
receive query refinement text;
generate a score for each of the search results based on the query refinement text, the search input, and the search results themselves; and
refine or reorder the search results based on the scores.
19 . A computing device comprising:
a display screen;
a processor; and
a memory coupled to the processor and configured to store an engineering application that utilizes one or more machine learning (ML) models, the engineering application when executed is operable to:
receive search input from a user,
extract from the search input a plurality of regions that each correspond to a different mode of input of a plurality of modes of input, wherein the plurality of regions of the search input includes at least one region that corresponds to schematic input or computer aided design (CAD) model input, and at least one other region that corresponds to text input or image input,
generate, using the one more ML models, one or more query embeddings that represent a semantic significance of input in each of the plurality of regions, the one or more ML models to produce first query embeddings that represent the semantic significance of the schematic input or CAD model input and second query embeddings that represent the semantic significance of text input or image,
compare the one or more query embeddings and a plurality of document embeddings of an embedding database that each correspond to a respective engineering document maintained in a library of engineering documents, and select one or more engineering documents having closest document embeddings, and
display search results based on the selected one or more engineering documents.
20 . The computing device of claim 19 , wherein the first and second query embeddings are temporary query embeddings, and the engineering application when executed is further operable to:
generate for each of the plurality of regions a temporary query embedding that represents a semantic significance of input in the respective region to produce a set of temporary query embeddings;
aggregate two or more of the set of temporary query embeddings to produce a main query embedding; and
use the main query embedding in the one or more query embeddings.