Cloud based data loss prevention system
A cloud based data loss prevention (DLP) system implements a split computing architecture using separate indexer system and detection system to perform indexing and data loss prevention monitoring. The cloud DLP system includes a computing system deployed outside of the enterprise data network and including a first computing cluster and a second computing cluster. The first computing cluster includes an indexer system to generate a search index from a pre-index containing hash values of structured data to be protected. The second computing cluster comprises a detection system configured to receive the search index and network data content, to apply a forward hash function based on a key to the network data content, and to detect in the hash values of the network data content for matching data in the search index and to generate an alert in response to matched data content being found in the network data content.
1. A system for providing data loss prevention services to an enterprise operating an enterprise data network, the system comprising:
a computing system deployed outside of the enterprise data network, the computing system comprising a first computing cluster having at least one hardware processor and a second computing cluster having at least one hardware processor,
wherein the first computing cluster comprises an indexer system configured to receive a pre-index containing first hash values of known non-malicious structured data formed using a forward hash function and a key and to generate a search index based on the pre-index, the search index being stored at a remote storage site; and
wherein the second computing cluster comprises a detection system configured to receive the search index from the remote storage site, to receive network data content being sent from or being sent to the enterprise, to apply the forward hash function based on the key to the network data content to form second hash values, to determine that data that should not be outside the enterprise is outside the enterprise when one or more of the second hash values match one or more of the first hash values using the search index, and to generate an alert indicating data that should not be outside the enterprise is outside the enterprise in response to determining that a match exists.
2. The system of claim 1 , wherein the pre-index is stored at the remote storage site in which the search index is stored.
3. The system of claim 2 , wherein the pre-index and the search index are stored at a multi-tenant cloud storage service.
4. The system of claim 1 , further comprising:
a client-side component deployed inside the enterprise data network, the client-side component being configured to generate the pre-index from the non-malicious structured data, the client-side component applying the forward hash function based on the key to generate the first hash values, the pre-index also comprising third hash values.
5. The system of claim 4 , wherein the client-side component is further configured to transmit the pre-index to a remote storage site outside of the enterprise data network.
6. The system of claim 4 , wherein the structured data comprises data in a tabular format and the pre-index comprises hash values of the data in the tabular format.
7. The system of claim 1 , wherein the network data content comprises network traffic data egress from or ingress to the enterprise data network and belonging to the enterprise and stored on one or more cloud service providers.
8. The system of claim 7 , wherein the key used in conjunction with the forward hash function to encode the pre-index is stored on the enterprise data network and the computing system, the key not being distributed to any of the one or more cloud service providers having data belonging to the enterprise stored thereon.
9. The system of claim 1 , wherein the detection system is further configured to apply remediation measures in response to determining that a match exists.
10. The system of claim 1 , wherein the detection system comprises a memory and a disk storage, the detection system partitioning the search index into a sub-index and storing the sub-index in the memory.
11. The system of claim 10 , wherein the detection system partitions the search index into the sub-index and a remainder index, the sub-index being stored in the memory and the remainder index being stored in the disk storage.
12. The system of claim 10 , wherein the detection system stores the search index in the disk storage.
13. The system of claim 10 , wherein the detection system applies a Bloom filter to the sub-index stored in the memory.