Systems and methods for discovery, classification, and indexing of data in a native computing system
In general, various aspects provide methods, apparatuses, systems, computing devices, computing entities, and/or the like for performing data discovery on a target computing system. In various aspects, a third party computing connects, via a public data network, to an edge node of the target computing system and instructs the target computing system to execute jobs to discover target data stored in data repositories in a private data network in the target computing system. In some aspects, the third party computing system may schedule the jobs on the target computing system based on computing resource availability on the target computing system.
1. A target computing system comprising:
one or more computing devices having access to data at a plurality of data sources on a private data network; at least one processing device of an external-facing subsystem, wherein the external-facing subsystem facilitates a connection between the private data network and a third-party computing data discovery system; and
a non-transitory computer-readable medium comprising instructions that, when executed by the at least one processing device of the external-facing subsystem, cause the at least one processing device to perform operations comprising:
deploying executable code comprising a neural network based classification model, of the third-party computing data discovery system, to the one or more computing devices corresponding to the plurality of data sources on the private data network to perform scanning and classification operations at the plurality of data sources utilizing the neural network based classification model of the external-facing subsystem without accessing the data at the plurality of data sources on the private data network;
causing the one or more computing devices corresponding to the plurality of data sources to execute the executable code of the external-facing subsystem to scan the plurality of data sources for target data;
causing the one or more computing devices corresponding to the plurality of data sources to execute the executable code of the external-facing subsystem to utilize the neural network based classification model with the target data to generate data type predictions indicating data type labels for the target data by:
generating a prediction score of a particular data type for a data item of the target data based on tokenized data or labelled data corresponding to the target data; and
identifying the particular data type as a data type label for the data item of the target data based on the prediction score satisfying a confidence score threshold; and
responsive to scanning and classifying the target data stored on the plurality of data sources, generating and storing metadata for the plurality of data sources, the metadata indicating types of the target data.
2. The target computing system of claim 1 , wherein deploying the executable code to the one or more computing devices comprises:
executing a client application to expose online functions to a third-party system; and
deploying, utilizing the client application, the executable code to the one or more computing devices to perform the scanning and classification operations in connection with one or more operations at the third-party system.
3. The target computing system of claim 2 , wherein causing the one or more computing devices to execute the executable code to scan the plurality of data sources for the target data comprises:
generating a job schedule comprising a plurality of jobs based on computing resource data of the one or more computing devices; and
causing the one or more computing devices to execute the executable code to scan the plurality of data sources according to the plurality of jobs in the job schedule and the computing resource data of the one or more computing devices.
4. The target computing system of claim 3 , wherein causing the one or more computing devices to execute the executable code to scan the plurality of data sources for the target data comprises causing the one or more computing devices to execute one or more scripts associated with each job of the plurality of jobs of the job schedule.
5. The target computing system of claim 3 , wherein generating the job schedule comprises scheduling the plurality of jobs to limit a number of simultaneous jobs at the one or more computing devices to a particular number of simultaneous jobs according to the computing resource data.
6. The target computing system of claim 3 , wherein the operations further comprise determining the computing resource data by determining current resource usage by the one or more computing devices and total available resources of the one or more computing devices.
7. The target computing system of claim 1 , wherein causing the one or more computing devices to execute the executable code to generate the data type predictions for the target data comprises causing the one or more computing devices to generate the tokenized data by tokenizing the data of the plurality of data sources.
8. The target computing system of claim 7 , wherein causing the one or more computing devices to generate the data type predictions indicating the data type labels comprises causing the one or more computing devices to:
generate labelled data for the data item by labeling the tokenized data; and
identify the data type label for the data item by generating the data type predictions for the target data based on the labelled data.
9. The target computing system of claim 1 , wherein generating and storing the metadata for the plurality of data sources comprises:
generating metadata comprising a predicted data type label for the data item of the target data and a storage location of the data item of the target data at a data source of the plurality of data sources; and
storing the metadata comprising a predicted data type label for the data item of the target data and the storage location of the data item of the target data at the data source of the plurality of data sources.
10. A method comprising:
deploying, by an external-facing subsystem of a target computing system, executable code comprising a neural network based classification model, of a third-party computing data discovery system, to one or more computing devices corresponding to a plurality of data sources on a private data network of the target computing system having access to data at a plurality of data sources of the target computing system on the private data network to perform scanning and classification operations at the plurality of data sources utilizing the neural network based classification model of the external-facing subsystem without accessing the data at the plurality of data sources on the private data network, wherein the external-facing subsystem facilitates a connection between the private data network and the third-party computing data discovery system;
generating, by the external-facing subsystem, a job schedule comprising a plurality of jobs corresponding to a plurality of data items at the plurality of data sources;
causing, by the external-facing subsystem, the one or more computing devices of the target computing system corresponding to the plurality of data sources to execute the executable code of the external-facing subsystem to scan the plurality of data sources for target data according to the plurality of jobs of the job schedule;
causing, by the external-facing subsystem, the one or more computing devices of the target computing system corresponding to the plurality of data sources to execute the executable code of the external-facing subsystem to utilize the neural network based classification model with the target data to generate classifications of data types for the target data at the plurality of data sources by:
generating a prediction score of a particular data type for a data item of the target data based on tokenized data or labelled data corresponding to the target data; and
identifying a particular data type as a data type label for the data item of the target data based on the prediction score satisfying a confidence score threshold; and
responsive to scanning and classifying the target data stored on the plurality of data sources, generating and storing metadata for the plurality of data sources, the metadata indicating types of the target data and storage locations of the target data at the plurality of data sources.
11. The method of claim 10 , further comprising:
responsive to a request to query the plurality of data sources in connection with a data subject access request, accessing the metadata for the plurality of data sources to identify a subset of the plurality of data sources that store a subset of the target data corresponding to the data subject access request; and
causing, by the external-facing subsystem, execution of processing operations to retrieve the subset of the target data from the subset of the plurality of data sources storing the subset of the target data.
12. The method of claim 10 , wherein deploying the executable code to the one or more computing devices comprises:
executing a client application that exposes online functions of the target computing system to the third-party computing data discovery system; and
deploying, utilizing the client application, the executable code to the one or more computing devices to perform the scanning and classification operations in connection with one or more operations at the third-party computing data discovery system.
13. The method of claim 10 , wherein causing the one or more computing devices of the target computing system to execute the executable code to scan the plurality of data sources for the target data comprises executing one or more scripts associated with each job of the plurality of jobs of the job schedule.
14. The method of claim 13 , wherein generating the job schedule comprises:
determining computing resource data comprising current resource usage by the one or more computing devices of the target computing system and total available resources of the one or more computing devices of the target computing system; and
scheduling the plurality of jobs to limit a number of simultaneous jobs at the one or more computing devices to a particular number of simultaneous jobs according to the computing resource data.
15. The method of claim 10 , wherein causing the one or more computing devices of the target computing system to execute the executable code to generate the classifications of the data types for the target data comprises:
causing the one or more computing devices to generate tokenized data by tokenizing the data of the plurality of data sources;
causing the one or more computing devices to generate labelled data by labeling the tokenized data; and
causing the one or more computing devices to generate the prediction score of the particular data type for the data item based on the labelled data corresponding to the target data.
16. The method of claim 10 , wherein causing the one or more computing devices to execute the executable code to scan the plurality of data sources for the target data according to the plurality of jobs of the job schedule comprises causing the one or more computing devices to schedule a scanning operation during at least one of a particular time period, a particular time of day, or during a particular day.
17. A non-transitory computer-readable medium comprising instructions that, when executed by computing hardware of an external-facing subsystem of a target computing system, cause the computing hardware to:
deploy executable code comprising a neural network based classification model, of a third-party computing data discovery system, to one or more computing devices corresponding to a plurality of data sources on a private data network of the target computing system having access to data at a plurality of data sources of the target computing system on the private data network to perform scanning and classification operations at the plurality of data sources utilizing the neural network based classification model of the external-facing subsystem without accessing the data at the plurality of data sources on the private data network, wherein the external-facing subsystem facilitates a connection between the private data network and the third-party computing data discovery system;
cause the one or more computing devices of the target computing system to execute the executable code to scan the plurality of data sources for target data;
cause the one or more computing devices of the target computing system to execute the executable code to utilize the neural network based classification model with the target data to generate classifications of data types for the target data at the plurality of data sources by:
generating a prediction score of a particular data type for a data item of the target data based on tokenized data or labelled data corresponding to the target data; and
identifying a particular data type as a data type label for the data item of the target data based on the prediction score satisfying a confidence score threshold; and
responsive to scanning and classifying the target data stored on the plurality of data sources, generate and store, for data source of the plurality of data sources, metadata indicating types and storage locations of the target data at the data source of the plurality of data sources.
18. The non-transitory computer-readable medium of claim 17 , further comprising instructions that, when executed by the computing hardware, cause the computing hardware to cause the one or more computing devices of the target computing system to execute the executable code to scan the plurality of data sources for the target data by:
generating a job schedule comprising a plurality of jobs based on computing resource data of the one or more computing devices;
determining current resource usage by the one or more computing devices of the target computing system and total available resources of the one or more computing devices of the target computing system; and
causing the one or more computing devices to execute the executable code to scan the plurality of data sources according to the plurality of jobs in the job schedule and the computing resource data.
19. The non-transitory computer-readable medium of claim 17 , further comprising instructions that, when executed by the computing hardware, cause the computing hardware to cause the one or more computing devices of the target computing system to execute the executable code to generate the classifications of the data types for the target data at the plurality of data sources by:
causing the one or more computing devices to generate tokenized data by tokenizing the data of the plurality of data sources; and
causing the one or more computing devices to generate, utilizing the neural network based classification model, the prediction score of the particular data type for the data item based on the tokenized data.
20. The non-transitory computer-readable medium of claim 17 , further comprising instructions that, when executed by the computing hardware, cause the computing hardware to:
responsive to a request to query the plurality of data sources in connection with a data subject access request, access the metadata for the plurality of data sources to identify a subset of the plurality of data sources that store a subset of the target data corresponding to the data subject access request; and
cause execution of processing operations to retrieve the subset of the target data from the subset of the plurality of data sources storing the subset of the target data.