Detecting personally identifiable information in data associated with a cloud computing system
Methods and systems for detecting personally identifiable information in data associated with a cloud computing system are described. An example method includes ingesting the data associated with the cloud computing system to generate source data. The method includes processing the source data by: performing cell-based de-duplication to generate cell-based de-duplicated data, subjecting the cell-based de-duplicated data to regular expression classification to generate a first subset of initial results, tokenizing the cell-based de-duplicated data to generate tokenized data, and de-duplicating the tokenized data and subjecting de-duplicated tokenized data to a first named entity recognition classification to generate a second subset of the initial results. The method includes cross-referencing the cell-based de-duplicated data and the initial results and subjecting output of the cross-referencing to a second named entity recognition classification to generate final results. The method includes processing the final results to detect any personally identifiable information in the final results.
1. A method implemented by a processing system, including at least one processor, the method comprising:
ingesting telemetry data associated with a cloud computing system to generate source data;
after the ingesting, as part of a first pass, using the at least one processor, processing the source data by:
performing cell-based de-duplication to generate cell-based de-duplicated data,
subjecting the cell-based de-duplicated data to regular expression classification to generate a first subset of initial results,
tokenizing the cell-based de-duplicated data to generate tokenized data, and
de-duplicating the tokenized data and subjecting de-duplicated tokenized data to a first named entity recognition classification to generate a second subset of the initial results;
after performing the first pass, as part of a second pass, cross-referencing the cell-based de-duplicated data by comparing the cell-based de-duplicated data with both of the first subset of the initial results and the second subset of the initial results, and subjecting output of the cross-referencing to a second named entity recognition classification to generate final results; and
using the at least one processor processing the final results to detect any personally identifiable information in the final results.
2. The method of claim 1 , wherein the source data comprises indexed columns.
3. The method of claim 1 , further comprising after de-duplicating the tokenized data, filtering the de-duplicated tokenized data before subjecting the de-duplicated tokenized data to the first named entity recognition classification to generate the second subset of the initial results.
4. The method of claim 1 , wherein processing the final results to detect any personally identifiable information in the final results comprises applying value rules and context rules to the final results.
5. The method of claim 1 , further comprising grouping any detected personally identifiable information by service operation names, activity names, or activity identifiers.
6. The method of claim 1 , wherein the final results include pointers to services responsible for any leaks of the personally identifiable information and pointers to source data where the personally identifiable information originated from.
7. The method of claim 1 , wherein the data associated with the cloud computing system comprises telemetry and data obtained by performing queries on any services or applications running in the cloud computing system.
8. A processing system, including at least one processor, the processing system comprising instructions that, when executed by the at least one processor, perform operations comprising:
ingest telemetry data associated with a cloud computing system to generate source data;
after ingesting, as part of a first pass, using the at least one processor, process the source data by operations including:
perform cell-based de-duplication to generate cell-based de-duplicated data,
subject the cell-based de-duplicated data to regular expression classification to generate a first subset of initial results,
tokenize the cell-based de-duplicated data to generate tokenized data, and
de-duplicate the tokenized data and subject de-duplicated tokenized data to a first named entity recognition classification to generate a second subset of the initial results;
after performing the first pass, as part of a second pass, cross-reference the cell-based de-duplicated data by comparing the cell-based de-duplicated data with both of the first subset of the initial results and the second subset of the initial results, and subject output of the cross-reference operation to a second named entity recognition classification to generate final results; and
using the at least one processor, process the final results to detect any personally identifiable information in the final results.
9. The processing system of claim 8 , wherein the source data comprises indexed columns.
10. The processing system of claim 8 , wherein the operations further comprise an operation to filter the de-duplicated tokenized data before subjecting the de-duplicated tokenized data to the first named entity recognition classification to generate the second subset of the initial results.
11. The processing system of claim 8 , wherein the operation to process the final results to detect any personally identifiable information in the final results comprises an operation to apply value rules and context rules to the final results.
12. The processing system of claim 8 , wherein the operations further comprise an operation to group any detected personally identifiable information by service operation names, activity names, or activity identifiers.
13. The processing system of claim 8 , wherein the final results include pointers to services responsible for any leaks of the personally identifiable information and pointers to source data where the personally identifiable information originated from.
14. The processing system of claim 8 , wherein the data associated with the cloud computing system comprises telemetry and data obtained by performing queries on any services or applications running in the cloud computing system.
15. A method implemented by a processing system, including at least one processor, the method comprising:
ingesting telemetry associated with a cloud computing system and generating batches of source data based on ingested telemetry, wherein each batch is identifiable using an associated batch identifier;
as part of a first pass, using the at least one processor, processing each batch of source data by:
performing cell-based deduplication to generate cell-based de-duplicated data,
subjecting the cell-based de-duplicated data to regular expression classification to generate a first subset of initial results,
tokenizing the cell-based de-duplicated data to generate tokenized data, and
de-duplicating and filtering the tokenized data to generate de-duplicated tokenized data and subjecting the de-duplicated tokenized data to a first named entity recognition classification to generate a second subset of the initial results;
after performing the first pass, as part of a second pass, cross-referencing the cell-based de-duplicated data by comparing the cell-based de-duplicated data with both of the first subset of the initial results and the second subset of the initial results, and subjecting output of the cross-referencing to a second named entity recognition classification to generate final results; and
using the at least one processor processing the final results to detect any personally identifiable information (PII) in the final results.
16. The method of claim 15 , wherein the source data comprises indexed columns.
17. The method of claim 15 , wherein processing the final results to detect any personally identifiable information in the final results comprises applying value rules and context rules to the final results.
18. The method of claim 15 , further comprising grouping any detected personally identifiable information by service operation names, activity names, or activity identifiers.
19. The method of claim 15 , wherein the final results include pointers to services responsible for any leaks of the personally identifiable information and pointers to source data where the personally identifiable information originated from.
20. The method of claim 15 , wherein the data associated with the cloud computing system further comprises data obtained by performing queries on any services or applications running in the cloud computing system.