DOCUMENT PROCESSING AND RESPONSE GENERATION SYSTEM
Disclosed are methods, systems, devices, apparatus, media, design structures, and other implementations, including a method that includes receiving, at a local device from a remote device, query data representative of a question relating to source content of a source document of a repository of a plurality of source documents, with the source content being associated with transformed content accessible from the local device. The method further includes generating, in response to determination, at the local device, of a match between the query data and at least one portion of the transformed content, output data comprising one or more of a pointer to access, in the source document accessible from the remote device, at least one portion of the source document corresponding to the at least one portion of the transformed content, or a copy of the at least one portion of the source document.
1 . A method comprising:
receiving, at a local device from a remote device, query data representative of a question relating to source content of a source document of a repository of a plurality of source documents, wherein the source content is associated with transformed content accessible from the local device; and
in response to determination, at the local device, of a match between the query data and at least one portion of the transformed content, generating output data comprising one or more of a pointer to access, in the source document accessible from the remote device, at least one portion of the source document corresponding to the at least one portion of the transformed content, or a copy of the at least one portion of the source document.
2 . The method of claim 1 , wherein the pointer to access the at least one portion of the source document comprises one or more of: document identifier identifying the source document in the repository of a plurality of source documents, network address of the source document in the repository of a plurality of source documents, starting and end locations of the at least one portion of the source document within the source document, or a word span.
3 . The method of claim 1 , wherein the at least one portion of the transformed content is stored in a repository of document object model (DOM) data items, wherein a DOM data item corresponding to the at least one portion of the transformed content comprises the pointer to access the at least one portion of the source document, and one or more of: at least one transformed element representative of the at least one portion of the source content, the at least one portion of the source content, or contextual information associated with the at least one portion of the source content.
4 . The method of claim 3 , wherein the contextual information associated with the at least one portion of the source document comprises feedback data from one or more users indicating responsiveness level of the output data to the question relating to the source content of the source document.
5 . The method of claim 1 , further comprising:
receiving a copy of the source document at the local device;
segmenting the copy of the source document into a plurality of document segments; and
transforming each of the plurality of document segments into one or more respective transformed segments according to one or more transformations.
6 . The method of claim 5 , further comprising:
receiving from an administrator information identifying location of the source document in the repository of the plurality of source documents.
7 . The method of claim 5 , wherein the one or more transformations comprise one or more of: a coarse linearization transform to generate coarse numerical vectors representative of coarse content of the plurality of document segments, or a fine-detail transformation to generate fine-detail transformed content records representative of the content of the plurality of document segments.
8 . The method of claim 7 , wherein at least one of the coarse transform and the fine-detail transformation comprises a transformation, applied to one or more document segments, based on Bidirectional Encoder Representations from Transformers (BERT) processing.
9 . The method of claim 7 , further comprising:
transforming the query data into transformed query data compatible with the transformed source content.
10 . The method of claim 9 , further comprising:
identifying one or more candidate portions in the transformed source content with respective coarse numerical vectors matching, according to a first criterion, the transformed query data.
11 . The method of claim 10 , further comprising:
determining, from one or more fine-detail transformed content records corresponding to the one or more candidate portions identified based on their respective numerical vectors, at least one fine-detail transformed content record matching, according to a second criterion, a fine-detail transformed data of the query data.
12 . The method of claim 9 , further comprising:
identifying one or more candidate portions in the transformed source content with respective fine-detail transformed content records matching, according to a second criterion, the transformed query data.
13 . The method of claim 5 , wherein segmenting the source content into the plurality of document segments comprises:
segmenting the source content into the plurality of document segments according to one or more pre-processing rules.
14 . The method of claim 13 , wherein segmenting the source content comprises:
segmenting the source content into the plurality of document segments according to hierarchical rules semantically associating one portion of the source content with one or more other portions of the source content.
15 . The method of claim 5 , further comprising:
retrieving augmented data associated with the received source document; and
segmenting a combination of the source document and the augmented data into the plurality of document segments.
16 . The method of claim 15 , wherein retrieving the augmented data comprises one or more of:
retrieving the augmented data based on links included in the source document; or
performing co-reference resolution to determine one or more names or expressions identified in the source document, and searching information repositories accessible from the local device to retrieve additional information associated with the determined one or more names or expressions identified in the source document.
17 . The method of claim 1 , further comprising:
identifying based on the received query data representative of the question one or more additional questions, from a repository of query data representative of a plurality of questions associated with source documents, to apply to the transformed content.
18 . The method of claim 1 , further comprising:
determining whether the received query data matches one of pre-determined questions; and
in response to determining that the received query data matches one of the pre-determined questions, generating the output data based on one or more answer data records associated with the matched one of the pre-determined questions.
19 . The method of claim 1 , wherein the local device is located at one of: a first network operated by a first party with the first network being different from a second network on which the remote device is located with the second network being operated by a second party, or a third network on which the remote device is also located.
20 . The method of claim 1 , further comprising:
segmenting, at the remote device, the copy of the source document into a plurality of document segments;
transforming, at the remote device, each of the plurality of document segments into one or more respective transformed segments according to one or more transformations; and
communicating the one or more respective transformed segments to the local device for storage at the local device.
21 . A system comprising:
one or more communication interfaces configured to receive, from a remote device, query data representative of a question relating to source content of a source document of a repository of a plurality of source documents, wherein the source content is associated with transformed content accessible from the system; and
a query engine configured to generate, in response to determination of a match between the query data and at least one portion of the transformed content, output data comprising one or more of a pointer to access, in the source document accessible from the remote device, at least one portion of the source document corresponding to the at least one portion of the transformed content, or a copy of the at least one portion of the source document.
22 . The system of claim 21 , further comprising:
an ingestion engine configured to:
segment a copy of the source document, received via the one or more interfaces, into a plurality of document segments; and
transform each of the plurality of document segments into one or more respective transformed segments according to one or more transformations.
23 . The system of claim 22 , wherein the one or more transformations comprise one or more of: a coarse linearization transform to generate coarse numerical vectors representative of coarse content of the plurality of document segments, or a fine-detail transformation to generate fine-detail transformed content records representative of the content of the plurality of document segments.
24 . The system of claim 22 , further comprising:
a query processing module configured to:
transform the query data into transformed query data compatible with the transformed source content;
identify one or more candidate portions in the transformed source content with respective coarse numerical vectors matching, according to a first criterion, the transformed query data; and
determine, from one or more fine-detail transformed content records corresponding to the one or more candidate portions identified based on their respective numerical vectors, at least one fine-detail transformed content record matching, according to a second criterion, a fine-detail transformed data of the query data.
25 . The system of claim 22 , wherein the ingestion engine configured to segment the source content is configured to:
segment the source content into the plurality of document segments according to one or more pre-processing rules, the one or more pre-processing rules including one or more hierarchical rules semantically associating one portion of the source content with one or more other portions of the source content.
26 . The system of claim 22 , wherein the ingestion engine is further configured to:
retrieve augmented data associated with the received source document; and
segment a combination of the source document and the augmented data into the plurality of document segments.
27 . The system of claim 21 , wherein the query engine is further configured to:
identify based on the received query data representative of the question one or more additional questions, from a repository of query data representative of a plurality of questions associated with source documents, to apply to the transformed content.
28 . The system of claim 21 , wherein the query engine is further configured to:
determine whether the received query data matches one of pre-determined questions; and
generate, in response to determining that the received query data matches one of the pre-determined questions, the output data based on one or more answer data records associated with the matched one of the pre-determined questions.
29 . The system of claim 21 , further comprising the remote device, wherein the system is located at one of: a first network operated by a first party with the first network being different from a second network on which the remote device is located with the second network being operated by a second party, or a third network on which the remote device is also located.
30 . A non-transitory computer readable media programmed with instructions, executable on one or more processors of a computing system, to:
receive, at a local device from a remote device, query data representative of a question relating to source content of a source document of a repository of a plurality of source documents, wherein the source content is associated with transformed content accessible from the local device; and
generate output data, in response to determination, at the local device, of a match between the query data and at least one portion of the transformed content, comprising one or more of a pointer to access, in the source document accessible from the remote device, at least one portion of the source document corresponding to the at least one portion of the transformed content, or a copy of the at least one portion of the source document.