Multimodal table extraction and semantic search in a machine learning platform for structuring data in organizations
Systems, methods, and computer-readable media for generating responses to natural-language queries regarding items in unstructured documents are disclosed. An application instance that includes one or more machine learning models receives, from a subscriber computing system, a query and document comprising unstructured data. Based on the unstructured data, the application instance generates a searchable data structure using a machine learning model. A query response is generated by performing a semantic search on the searchable data structure. The query response is provided to a target application.
1. A provider computing system associated with a provider entity and comprising at least one processor, at least one memory, and one or more non-transitory computer readable media, excluding transitory signals, storing instructions, which when executed by the at least one processor, perform operations for generating responses to natural-language queries regarding items in unstructured documents, the operations comprising:
receiving, at an application instance communicatively coupled to a subscriber computing system of a plurality of subscriber computing systems, a query and a document comprising unstructured data;
performing pre-processing operations on at least a portion of the document comprising the unstructured data, the pre-processing operations comprising extracting, by a trained neural network, alphanumeric data from the unstructured data by generating a plurality of bounding boxes, each of the plurality of bounding boxes corresponding to a detected cell in a plurality of cells;
identifying, in the unstructured data, a particular image comprising a globally applicable item related to the alphanumeric data;
extracting, from the particular image, the globally applicable item;
generating a searchable data structure comprising the alphanumeric data stored relationally to the globally applicable item;
using the searchable data structure, performing operations comprising:
generating a query response by performing, by a semantic similarity model, a semantic search;
determining a set of coordinates in the document, wherein the set of coordinates corresponds to a query response item included in the alphanumeric data;
including the determined set of coordinates in the query response; and
transmitting query response to a target application operated or hosted at least in part by the subscriber computing system or a subscriber entity associated with the subscriber computing system.
2. The system of claim 1 , wherein the application instance is provided by the provider entity, and wherein the application instance is on a virtual network associated with the subscriber entity.
3. A method for generating responses to natural-language queries regarding items in unstructured documents, the method comprising:
receiving, at an application instance communicatively coupled to a subscriber computing system of a plurality of subscriber computing systems, a query and a document comprising unstructured data;
performing pre-processing operations on at least a portion of the document comprising the unstructured data, the pre-processing operations comprising generating an optimized model input comprising at least one parsed document section that includes alphanumeric data by:
using a computer vision machine learning model to:
detect a first image in the unstructured data, wherein the first image comprises the alphanumeric data;
generate a bounding box to encapsulate the first image; and
parse, from the first image, a second image comprising the alphanumeric data, the second image defined by the bounding box; and
extracting, by a trained machine learning model, the alphanumeric data from the second image;
identifying, in the unstructured data, a particular image comprising a globally applicable item related to the alphanumeric data;
extracting, from the particular image, the globally applicable item;
generating a searchable data structure comprising the alphanumeric data stored relationally to the globally applicable item; and
for each of the at least one parsed document section and using the searchable data structure,
generating a query response by performing, by a semantic similarity model, a semantic search; and
transmitting the query response to a target application operated or hosted at least in part by the subscriber computing system or a subscriber entity associated with the subscriber computing system.
4. The method of claim 3 , wherein the searchable data structure comprises a key-value pair.
5. The method of claim 3 ,
wherein the first image comprises a table; and
wherein extracting the alphanumeric data comprises generating, by a trained neural network, a plurality of bounding boxes, each corresponding to a detected cell in a plurality of cells from the table.
6. The method of claim 5 ,
wherein the trained neural network is a convolutional neural network (CNN).
7. The method of claim 5 , wherein performing the semantic search comprises:
using the searchable data structure, determining, by the trained neural network, a context for the query, the context relating to a value included in the searchable data structure; and
generating, using the determined context for the query and the searchable data structure, at least one response option.
8. The method of claim 7 , further comprising:
determining a set of coordinates in the document, wherein the set of coordinates corresponds to the at least one response option; and
generating a second searchable data structure comprising the at least one response option stored relationally to the determined set of coordinates.
9. The method of claim 7 , further comprising:
generating a sentence comprising the at least one response option; and
providing the generated sentence via an output device associated with the subscriber computing system.
10. The method of claim 5 ,
wherein the searchable data structure stores the globally applicable item relationally to each row in the alphanumeric data.
11. The method of claim 3 , wherein the globally applicable item comprises an image.
12. The method of claim 3 , wherein the globally applicable item comprises alphanumeric information.
13. The method of claim 3 , wherein extracting the alphanumeric data comprises:
determining a plurality of coordinate sets, each of the coordinate sets associated with a token included in the second image; and
based on token proximity determined using the coordinate sets, linking a first token and a second token included in the second image,
wherein at least the first token and the second token are ordered in phrasal form.
14. The method of claim 3 , wherein generating optimized model input comprises applying a domain-specific ontology to at least one of the document and the optimized model input.
15. The method of claim 14 , wherein the domain-specific ontology relates to at least one of an insurance policy term, medical information, or a medication.
16. The method of claim 3 , wherein the subscriber computing system and the target application are provided by the subscriber entity, and wherein the application instance is provided by a provider entity different from the subscriber entity.
17. The method of claim 3 , wherein the application instance is on a virtual network associated with the subscriber entity.
18. The method of claim 3 , wherein generating optimized model input comprises determining a type of output needed based on at least one of: a previously stored setting, a subscriber-defined runtime parameter or a feature of the target application.
19. One or more non-transitory computer readable media excluding transitory signals, the media storing instructions, which when executed by at least one processor, perform operations for generating responses to natural-language queries regarding items in unstructured documents, the operations comprising:
receiving, at an application instance communicatively coupled to a subscriber computing system of a plurality of subscriber computing systems, a query and a document comprising unstructured data;
performing pre-processing operations on at least a portion of the document comprising the unstructured data, the pre-processing operations comprising generating an optimized model input comprising at least one parsed document section that includes alphanumeric data by:
using a computer vision machine learning model,
detecting a first image in the unstructured data, wherein the first image comprises the alphanumeric data;
generating a bounding box to encapsulate the first image;
parsing, from the first image, a second image comprising the alphanumeric data, the second image defined by the bounding box; and
extracting, by a trained neural network, the alphanumeric data from the second image;
identifying, in the unstructured data, a particular image comprising a globally applicable item related to the alphanumeric data;
extracting, from the particular image, the globally applicable item;
generating a searchable data structure comprising the alphanumeric data stored relationally to the globally applicable item; and
for each of the at least one parsed document section and using the searchable data structure,
generating a query response by performing, by a semantic similarity model, a semantic search; and
transmitting the query response to a target application operated or hosted at least in part by the subscriber computing system or a subscriber entity associated with the subscriber computing system.
20. The media of claim 19 , wherein the application instance is provided by a provider entity different from the subscriber entity, and wherein the application instance is on a virtual network associated with the subscriber entity.