Method for maintaining privacy and security of data

1. A method, comprising:

receiving, in parallel by each of a plurality of datasource processors, datasource objects from respective ones of a plurality of datasources, each datasource processor being communicatively coupled to one or more of the plurality of datasources;

communicating, by the plurality of datasource processors, with an artificial intelligence (AI) engine to generate an object summary for each of the datasource objects, wherein each respective object summary includes an object identifier, one or more local entities derived from the respective datasource object, and a mapping from each of the one or more local entities to one or more attributes derived from the respective datasource object;

generating the object summary for each of the datasource objects, wherein generating the object summary for each datasource object comprises:

determining, by a text extractor of the AI engine, text included in the datasource object;

determining, by a layout extractor of the AI engine, a layout of the text;

arranging, by the layout extractor of the AI engine, the text based on the layout of the text;

determining, by an object classifier of the AI engine, a category of the datasource object based on the text and layout of the text;

determining, by an attribute extractor of the AI engine, attributes included in the datasource object; and

grouping, by a local entity resolver of the AI engine, the attributes into one or more attribute groups;

augmenting each of the object summaries to include information regarding global entities by (i) aggregating the local entities in batches at a queue of a global entity resolver, and (ii) for each of the local entities aggregated at the queue, mapping by the global entity resolver the local entity to one of the global entities;

evaluating, by each of a plurality of policy engines, a subset of the object summaries with respect to a security and/or privacy policy, wherein the evaluation includes determining, for each object summary, whether a security and/or privacy violation exists with respect to the respective datasource object with which the object summary is associated, wherein the policy engines perform the evaluation step in parallel with one another, and wherein the security and/or privacy violation is recognized when an operation concerning the datasource object associated with the object summary violates the security and/or privacy policy; and

when the security and/or privacy violation is recognized, applying a remediation measure in connection with the datasource object for which the security and/or privacy violation exists.

2. The method of claim 1 , wherein the text extractor determines text included in the datasource object by:

performing optical character recognition (OCR) on the datasource object to produce OCR output;

determining whether a quality of the OCR output is acceptable; and

when the quality of the OCR output is determined to be acceptable, returning the text present in the OCR output.

3. The method of claim 1 , wherein the text extractor determines text included in the datasource object by:

determining whether the datasource object contains personal identifiable information (PII); and

converting the datasource object from a non-portable document format to a portable document format (PDF) only when the datasource object contains PII.

4. The method of claim 1 , wherein the object classifier determines the category of the datasource object by:

determining whether the datasource object matches one of a plurality of templates;

when the datasource object matches one of the templates, determining the category of the datasource object based on the matched template;

otherwise, when the datasource object does not match any of the templates, determining whether a number of words of the datasource object is below a predetermined threshold;

when the number of words of the datasource object is below the predetermined threshold, determining the category of datasource object based on a bidirectional encoder representations from transformers (BERT) language model; and

otherwise, when the number of words of the datasource object is not below the predetermined threshold, determining the category of datasource object based on a term frequency inverse document frequency (TFIDF) language model.

5. The method of claim 1 , wherein the attribute extractor determines attributes included in the datasource object by:

determining whether the datasource object matches one of a plurality of templates;

when the datasource object matches one of the templates, determining the attributes based on the matched template;

otherwise, when the datasource object does not match any of the templates, determining whether the datasource object contains any tables or forms;

when the datasource object contains any tables or forms, determining the attributes included in the datasource object using a structure attribute extractor that is configured to extract attributes from tables and forms;

otherwise, when the datasource object does not contain any tables or forms, determining whether the datasource object is an identity document;

when the datasource object is an identity document, determining the attributes included in the datasource object using a deep learning model configured to determine attributes from identity documents; and

otherwise, when the datasource object is not an identity document, determining the attributes included in the datasource object using a generic attribute extractor that is configured to determine attributes from unformatted text.

6. The method of claim 5 , wherein each of the attributes comprises a key, a value and a confidence level indicating a confidence associated with the key being paired with the value, wherein for each of the attributes that are not determined based on the matched template, determining whether refinement of the confidence level is needed, and when refinement of the confidence level is needed, adjusting the confidence level based on one or more of the layout of the text or a source of truth datastore.

7. The method of claim 1 , wherein generating the object summary for each datasource object comprises further comprises removing, by the local entity resolver, duplicate instances of the one or more attribute groups.

8. The method of claim 1 , wherein mapping each of the local entities to one of the global entities comprises:

in a first phase of resolution, determining whether the local entity is associated with any existing global entities, and when the local entity is associated with one of the existing global entities, mapping the local entity to the associated existing global entity, otherwise, leaving the local entity unmapped; and

in a second phase of resolution following the first phase, for the local entities that are unmapped, clustering one or more of the local entities having substantially similar attributes into clusters and assigning a new global entity to each of the clusters.

9. The method of claim 1 , further comprising:

in response to receiving an update to the security and/or privacy policy, updating the security and/or privacy policy;

selecting only ones of the object summaries that are impacted by the update to the security and/or privacy policy; and

evaluating the selected object summaries based on the updated security and/or privacy policy.

10. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to:

receive, in parallel by each of a plurality of datasource processors, datasource objects from respective ones of a plurality of datasources, each datasource processor being communicatively coupled to one or more of the plurality of datasources;

communicate, by the plurality of datasource processors, with an artificial intelligence (AI) engine to generate an object summary for each of the datasource objects, wherein each respective object summary includes an object identifier, one or more local entities derived from the respective datasource object, and a mapping from each of the one or more local entities to one or more attributes derived from the respective datasource object;

generate the object summary for each of the datasource objects, wherein generating the object summary for each datasource object comprises:

determining, by a text extractor of the AI engine, text included in the datasource object;

determining, by a layout extractor of the AI engine, a layout of the text;

arranging, by the layout extractor of the AI engine, the text based on the layout of the text;

determining, by an object classifier of the AI engine, a category of the datasource object based on the text and layout of the text;

determining, by an attribute extractor of the AI engine, attributes included in the datasource object; and

grouping, by a local entity resolver of the AI engine, the attributes into one or more attribute groups;

augment each of the object summaries to include information regarding global entities by (i) aggregating the local entities in batches at a queue of a global entity resolver, and (ii) for each of the local entities aggregated at the queue, mapping by the global entity resolver the local entity to one of the global entities;

evaluate, by each of a plurality of policy engines, a subset of the object summaries with respect to a security and/or privacy policy, wherein the evaluation includes determining, for each object summary, whether a security and/or privacy violation exists with respect to the respective datasource object with which the object summary is associated, wherein the policy engines perform the evaluation step in parallel with one another, and wherein the security and/or privacy violation is recognized when an operation concerning the datasource object associated with the object summary violates the security and/or privacy policy; and

when the security and/or privacy violation is recognized, apply a remediation measure in connection with the datasource object for which the security and/or privacy violation exists.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the text extractor determines text included in the datasource object by:

performing optical character recognition (OCR) on the datasource object to produce OCR output;

determining whether a quality of the OCR output is acceptable; and

when the quality of the OCR output is determined to be acceptable, storing the text present in the OCR output.

12. The non-transitory computer-readable storage medium of claim 10 , wherein the text extractor determines text included in the datasource object by:

determining whether the datasource object contains personal identifiable information (PII); and

converting the datasource object from a non-portable document format to a portable document format (PDF) only when the datasource object contains PII.

13. The non-transitory computer-readable storage medium of claim 10 , wherein the object classifier determines the category of the datasource object by:

determining whether the datasource object matches one of a plurality of templates;

when the datasource object matches one of the templates, determining the category of the datasource object based on the matched template;

otherwise, when the datasource object does not match any of the templates, determining whether a number of words of the datasource object is below a predetermined threshold;

when the number of words of the datasource object is below the predetermined threshold, determining the category of datasource object based on a bidirectional encoder representations from transformers (BERT) language model; and

otherwise, when the number of words of the datasource object is not below the predetermined threshold, determining the category of datasource object based on a term frequency inverse document frequency (TFIDF) language model.

14. The non-transitory computer-readable storage medium of claim 10 , wherein the attribute extractor determines attributes included in the datasource object by:

determining whether the datasource object matches one of a plurality of templates;

when the datasource object matches one of the templates, determining the attributes based on the matched template;

otherwise, when the datasource object does not match any of the templates, determining whether the datasource object contains any tables or forms;

when the datasource object contains any tables or forms, determining the attributes included in the datasource object using a structure attribute extractor that is configured to extract attributes from tables or forms;

otherwise, when the datasource object does not contain any tables or forms, determining whether the datasource object is an identity document;

when the datasource object is an identity document, determining the attributes included in the datasource object using a deep learning model configured to determine attributes from identity documents; and

otherwise, when the datasource object is not an identity document, determining the attributes included in the datasource object using a generic attribute extractor that is configured to determine attributes from unformatted text.

15. The non-transitory computer-readable storage medium of claim 14 , wherein each of the attributes comprises a key, a value and a confidence level indicating a confidence associated with the key being paired with the value, wherein for each of the attributes that are not determined based on the matched template, determining whether refinement of the confidence level is needed, and when refinement of the confidence level is needed, adjusting the confidence level based on one or more of the layout of the text or a source of truth datastore.

16. The non-transitory computer-readable storage medium of claim 10 , wherein generating the object summary for each datasource object comprises further comprises removing, by the local entity resolver, duplicate instances of the one or more attribute groups.

17. The non-transitory computer-readable storage medium of claim 10 , wherein mapping each of the local entities to one of the global entities comprises:

in a first phase of resolution, determining whether the local entity is associated with any existing global entities, and when the local entity is associated with one of the existing global entities, mapping the local entity to the associated existing global entity, otherwise, leaving the local entity unmapped; and

in a second phase of resolution following the first phase, for the local entities that are unmapped, clustering one or more of the local entities having substantially similar attributes into clusters and assigning a new global entity to each of the clusters.

18. The non-transitory computer-readable storage medium of claim 10 , further comprising instructions that cause the processor to:

in response to receiving an update to the security and/or privacy policy, update the security and/or privacy policy;

select only ones of the object summaries that are impacted by the update to the security and/or privacy policy; and

evaluate the selected object summaries based on the updated security and/or privacy policy.