Multimodal table extraction and semantic search in a machine learning platform for structuring data in organizations

1 . A provider computing system associated with a provider entity and comprising at least one processor, at least one memory, and one or more non-transitory computer readable media, excluding transitory signals, storing instructions, which when executed by the at least one processor, perform operations for generating responses to natural-language queries regarding items in unstructured documents, the operations comprising:

receiving, at an application instance communicatively coupled to a subscriber computing system of a plurality of subscriber computing systems, a query and a document comprising unstructured data;

performing pre-processing operations on at least a portion of the document comprising the unstructured data, the pre-processing operations comprising extracting, by a trained neural network, alphanumeric data from the unstructured data by generating a plurality of bounding boxes, each of the plurality of bounding boxes corresponding to a detected cell in a plurality of cells;

generating a searchable data structure comprising the extracted alphanumeric data;

using the searchable data structure, performing operations comprising:

identifying, in the unstructured data, an image comprising a globally applicable item related to the alphanumeric data;

extracting, from the image, the globally applicable item;

causing the searchable data structure to store the globally applicable item relationally to each row in the alphanumeric data;

generating a query response by performing, by a semantic similarity model, a semantic search using the globally applicable item or at least an additional portion of the searchable data structure;

determining a set of coordinates in the unstructured document, wherein the set of coordinates corresponds to a query response item included in the alphanumeric data;

including the determined set of coordinates in the query response; and

transmitting query response to a target application operated or hosted at least in part by the subscriber computing system or a subscriber entity associated with the subscriber computing system.

2 . The system of claim 1 , wherein the application instance is provided by the provider entity, and wherein the application instance is on a virtual network associated with the subscriber entity.

3 . A method for generating responses to natural-language queries regarding items in unstructured documents, the method comprising:

receiving, at an application instance communicatively coupled to a subscriber computing system of a plurality of subscriber computing systems, a query and a document comprising unstructured data;

performing pre-processing operations on at least a portion of the document comprising the unstructured data, the pre-processing operations comprising generating an optimized model input comprising extracting alphanumeric data from the at least a portion of the document;

generating a searchable data structure comprising the alphanumeric data; and

for each of the at least one parsed document section and using the searchable data structure,

identifying, in the unstructured data, an image comprising a globally applicable item related to the alphanumeric data;

extracting, from the image, the globally applicable item;

causing the searchable data structure to store the globally applicable item relationally to each row in the alphanumeric data;

generating a query response by performing, by a semantic similarity model, a semantic search using the globally applicable item or at least an additional portion of the searchable data structure; and

transmitting the query response to a target application operated or hosted at least in part by the subscriber computing system or a subscriber entity associated with the subscriber computing system.

4 . The method of claim 3 , wherein the searchable data structure comprises a key-value pair.

5 . The method of claim 3 ,

wherein the first image comprises a table; and

wherein extracting the alphanumeric data comprises generating, by a trained neural network, a plurality of bounding boxes, each corresponding to a detected cell in a plurality of cells from the table.

6 . The method of claim 5 ,

wherein the trained neural network is a convolutional neural network (CNN).

7 . The method of claim 5 , wherein performing the semantic search comprises:

using the searchable data structure, determining, by the trained neural network, a context for the query, the context relating to a value included in the searchable data structure; and

generating, using the determined context for the query and the searchable data structure, the at least one response option.

8 . The method of claim 7 , further comprising:

determining a set of coordinates in the unstructured document, wherein the set of coordinates corresponds to the at least one response option; and

generating a second searchable data structure comprising the response option stored relationally to the determined set of coordinates.

9 . The method of claim 7 , further comprising:

generating a sentence comprising the response option; and

providing the generated sentence via an output device associated with the subscriber computing system.

10 . The method of claim 3 , wherein the extracted globally applicable item comprises an image.

11 . The method of claim 3 , wherein the extracted globally applicable item comprises alphanumeric information.

12 . The method of claim 3 , wherein extracting the alphanumeric data comprises:

determining a plurality of coordinate sets, each of the coordinate sets associated with a token included in the second image; and

based on token proximity determined using the coordinate sets, linking a first token and a second token included in the second image,

wherein at least the first token and the second token are ordered in phrasal form via the linking.

13 . The method of claim 3 , wherein extracting alphanumeric data from the at least a portion of the document comprises:

using a computer vision machine learning model to:

detect an additional image in the unstructured data, wherein the additional image comprises the alphanumeric data;

generate a bounding box to encapsulate the additional image; and

parse, from the additional image, a second image comprising the alphanumeric data, the second image defined by the bounding box;

extracting, by a trained machine learning model, the alphanumeric data from the second image; and

generating the optimized model input by applying a domain-specific ontology to at least one of the document and the alphanumeric data.

14 . The method of claim 13 , wherein the domain-specific ontology relates to at least one of an insurance policy term, medical information, or a medication.

15 . The method of claim 3 , wherein the subscriber computing system and the target application are provided by the subscriber entity, and wherein the application instance is provided by a provider entity different from the subscriber entity.

16 . The method of claim 3 , wherein the application instance is on a virtual network associated with the subscriber entity.

17 . The method of claim 3 , wherein generating optimized model input comprises determining a type of output needed based on at least one of: a previously stored setting, a subscriber-defined runtime parameter or a feature of the target application.

18 . One or more non-transitory computer readable media excluding transitory signals, the media storing instructions, which when executed by at least one processor, perform operations for generating responses to natural-language queries regarding items in unstructured documents, the operations comprising:

receiving, at an application instance communicatively coupled to a subscriber computing system of a plurality of subscriber computing systems, a query and a document comprising unstructured data;

performing pre-processing operations on at least a portion of the document comprising the unstructured data, the pre-processing operations comprising generating an optimized model input comprising at least one parsed document section that includes alphanumeric data by:

using a computer vision machine learning model,

detecting a first image in the unstructured data, wherein the first image comprises the alphanumeric data;

generating a bounding box to encapsulate the first image;

parsing, from the first image, a second image comprising the alphanumeric data, the second image defined by the bounding box; and

extracting, by a trained neural network, the alphanumeric data from the second image;

generating a searchable data structure comprising the alphanumeric data; and

for each of the at least one parsed document section and using the searchable data structure,

identifying, in the unstructured data, an image comprising a globally applicable item related to the alphanumeric data;

extracting, from the image, the globally applicable item;

causing the searchable data structure to store the globally applicable item relationally to each row in the alphanumeric data;

generating a query response by performing, by a semantic similarity model, a semantic search using the globally applicable item or at least an additional portion of the searchable data structure; and

transmitting the query response to a target application operated or hosted at least in part by the subscriber computing system or a subscriber entity associated with the subscriber computing system.

19 . The media of claim 18 , wherein the application instance is provided by a provider entity different from the subscriber entity, and wherein the application instance is on a virtual network associated with the subscriber entity.