Automated processing of unstructured text data in paired data fields of a document
An apparatus comprises a processing device configured to select a first data field of a first type that is associated with a second data field of a second type in a document, to determine an embedding of terms of unstructured text data in the first data field and to identify a subset of paired data fields from an unstructured text database based at least in part on metrics characterizing similarity between (i) the embedding of terms in the first data field and (ii) embeddings of terms in data fields of the first type in the paired data fields. The processing device is further configured to determine syntactic differences between the unstructured text data in the first data field and the identified subset of paired data fields, and to provide recommendations for unstructured text data to fill the second data field in the document based on the syntactic differences.
1. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured to perform steps of:
selecting a first data field of a first type from a document, the first data field corresponding to a question and being associated with a second data field of a second type in the document, the second data field corresponding to an answer to the question;
determining an embedding of terms of unstructured text data in the first data field;
identifying, from an unstructured text database comprising a plurality of paired data fields each comprising a data field of the first type that is associated with a data field of the second type, a subset of the plurality of paired data fields based at least in part on a one or more metrics characterizing similarity between (i) the embedding of terms of the unstructured text data in the first data field and (ii) embeddings of terms of unstructured text data in the data fields of the first type in the plurality of paired data fields;
determining syntactic differences between the unstructured text data in the first data field of the document and the unstructured text data in the data fields of the first type in the identified subset of paired data fields to determine whether the unstructured text data fields of the second type in the identified subset are relevant to the question;
providing one or more recommendations for unstructured text data to fill the second data field in the document, the one or more recommendations comprising at least a given portion of unstructured text data in one or more of the data fields of the second type in the identified subset of paired data fields, the given portion of the unstructured text data being determined based at least in part on the syntactic differences between the unstructured text data in the first data field and the unstructured text data in the data fields of the first type in the identified subset of paired data fields;
receiving feedback regarding the one or more provided recommendations; and
updating, based on the received feedback, a weighting of at least one unstructured term in a supervised learning dataset iteratively improve recommendations for unstructured data.
2. The apparatus of claim 1 wherein the document comprises a questionnaire, and the plurality of paired data fields comprise a plurality of previously-answered question-answer pairs from one or more previously-completed questionnaires.
3. The apparatus of claim 1 wherein the terms of the unstructured text data of the first data field comprise words, and wherein determining the embedding of the terms of the unstructured text data of the first data field comprises computing a vector space embedding of the terms of the unstructured text data of the first data field.
4. The apparatus of claim 3 wherein each term of the unstructured text data of the first data field is a single word.
5. The apparatus of claim 3 wherein the vector space embedding comprises a numerical vector with entries representing weights for respective ones of the terms of the unstructured text data for the first data field, a given one of the weights comprising one or more of:
a term frequency-inverse document frequency weight component;
a logarithm of a term frequency weight component;
a bag-of-words weight component; and
an entropy weight component.
6. The apparatus of claim 1 wherein the one or more metrics characterizing similarity between (i) the embedding of the terms of the unstructured text data in the first data field and (ii) the embeddings of the terms of the unstructured text data in the data fields of the first type in the plurality of paired data fields comprise at least one of:
a cosine similarity metric;
a Sørensen-Dice similarity metric; and
a Jaccard similarity metric.
7. The apparatus of claim 1 wherein identifying the subset of paired data fields comprises identifying a designated number of the plurality of paired data fields from the unstructured text database with highest values for the one or more metrics characterizing similarity between (i) the embedding of the terms of the unstructured text data in the first data field and (ii) the embeddings of the terms of the unstructured text data in the data fields of the first type in the plurality of paired data fields.
8. The apparatus of claim 1 wherein determining the syntactic differences comprises computing at least one of an edit distance and a Hamming distance between the unstructured text data in the first data field and the unstructured text data in the data fields of the first type in the identified subset of paired data fields.
9. The apparatus of claim 1 wherein providing the one or more recommendations for unstructured text data to fill the second data field in the document comprises automatically filling at least one of the one or more recommendations in the second data field in the document.
10. The apparatus of claim 1 , the at least one processing device being further configured to perform steps of updating the supervised learning dataset by:
identifying a symmetric set-difference of the terms of the unstructured text data of the first data field and the terms of the unstructured text data in the data fields of the first type in the identified subset of paired data fields; and
updating a weighting of at least one unstructured term based on the received feedback includes modifying weights in embeddings of terms belonging to the identified symmetric set-difference relative to weights in embeddings of terms not belonging to the identified symmetric set-difference.
11. The apparatus of claim 1 , the at least one processing device being further configured to perform the step of updating the supervised learning dataset by, responsive to determining that a user accepts at least one of the one or more recommendations, adding the first data field and said at least one recommendation as a new paired data field in the unstructured text database.
12. The apparatus of claim 1 wherein providing the one or more recommendations for unstructured text data to fill the second data field in the document comprises providing to a user the identified subset of paired data fields and a representation of the between the unstructured text data in the first data field and the unstructured text data in the data fields of the first type in the identified subset of paired data fields.
13. The apparatus of claim 12 wherein providing the identified subset of paired data fields to the user comprises:
determining, for at least a given one of the identified subset of paired data fields, a first additional paired data field in the unstructured text database that has a data field of the second type with unstructured text data that is semantically similar to unstructured text data in a data field of the second type in the given paired data field;
providing the additional paired data field to the user; and
responsive to the user accepting at least a portion of the unstructured text data in the data field of the second type in the additional paired data field to fill the second data field, presenting to the user at least a second additional paired data field in the unstructured text database that has a data field of the second type with unstructured text data that is semantically similar to unstructured text data in a data field of the second type in the given paired data field.
14. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform steps of:
selecting a first data field of a first type from a document, the first data field corresponding to a question and being associated with a second data field of a second type in the document, the second data field corresponding to an answer to the question;
determining an embedding of terms of unstructured text data in the first data field;
identifying, from an unstructured text database comprising a plurality of paired data fields each comprising a data field of the first type that is associated with a data field of the second type, a subset of the plurality of paired data fields based at least in part on one or more metrics characterizing similarity between (i) the embedding of terms of the unstructured text data in the first data field and (ii) embeddings of terms of unstructured text data in the data fields of the first type in the plurality of paired data fields;
determining syntactic differences between the unstructured text data in the first data field of the document and the unstructured text data in the data fields of the first type in the identified subset of paired data fields to determine whether the unstructured text data fields of the second type in the identified subset are relevant to the question;
providing one or more recommendations for unstructured text data to fill the second data field in the document, the one or more recommendations comprising at least a given portion of unstructured text data in one or more of the data fields of the second type in the identified subset of paired data fields, the given portion of the unstructured text data being determined based at least in part on the syntactic differences between the unstructured text data in the first data field and the unstructured text data in the data fields of the first type in the identified subset of paired data fields;
receiving feedback regarding the one or more provided recommendations; and
updating, based on the received feedback, a weighting of at least one unstructured term in a supervised learning dataset to iteratively improve recommendations for unstructured data.
15. The computer program product of claim 14 wherein providing the one or more recommendations for unstructured text data to fill the second data field in the document comprises automatically filling at least one of the one or more recommendations in the second data field in the document.
16. A method comprising:
selecting a first data field of a first type from a document, the first data field corresponding to a question and being associated with a second data field of a second type in the document, the second data field corresponding to an answer to the question;
determining an embedding of terms of unstructured text data in the first data field;
identifying, from an unstructured text database comprising a plurality of paired data fields each comprising a data field of the first type that is associated with a data field of the second type, a subset of the plurality of paired data fields based at least in part on one or more metrics characterizing similarity between (i) the embedding of terms of the unstructured text data in the first data field and (ii) embeddings of terms of unstructured text data in the data fields of the first type in the plurality of paired data fields;
determining syntactic differences between the unstructured text data in the first data field of the document and the unstructured text data in the data fields of the first type in the identified subset of paired data fields to determine whether the unstructured text data fields of the second type in the identified subset are relevant to the question;
providing one or more recommendations for unstructured text data to fill the second data field in the document, the one or more recommendations comprising at least a given portion of unstructured text data in one or more of the data fields of the second type in the identified subset of paired data fields, the given portion of the unstructured text data being determined based at least in part on the syntactic differences between the unstructured text data in the first data field and the unstructured text data in the data fields of the first type in the identified subset of paired data fields;
receiving feedback regarding the one or more provided recommendations; and
updating, based on the received feedback, a weighting of at least one unstructured term in a supervised learning dataset to iteratively improve recommendations for unstructured data;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
17. The method of claim 16 wherein providing the one or more recommendations for unstructured text data to fill the second data field in the document comprises automatically filling at least one of the one or more recommendations in the second data field in the document.