Data loss prevention via indexed document matching
Systems and methods include obtaining a file to be checked for Data Loss Prevention (DLP); determining a cryptographic hash of the file and comparing the cryptographic hash to corresponding cryptographic hashes of indexed files; responsive to a match between the cryptographic hash and one of the corresponding cryptographic hashes, determining a DLP match and performing an action based thereon; responsive to no match, extracting text from the file and creating an ordered sequence of hashes of variable length chunks of the extracted text; and determining the DLP match with one of the indexed files based on comparing the ordered sequence of hashes with corresponding ordered sequence of hashes of the indexed files.
1. A non-transitory computer-readable storage medium having computer-readable code stored thereon for programming one or more processors to perform steps of:
obtaining a file to be checked for Data Loss Prevention (DLP);
determining a cryptographic hash of the file and comparing the cryptographic hash to corresponding cryptographic hashes of indexed files, wherein only fingerprints of the indexed files are shared to protect sensitive information;
responsive to a match between the cryptographic hash and one of the corresponding cryptographic hashes, determining a DLP match and performing an action based thereon;
responsive to no match, extracting text from the file and creating an ordered sequence of hashes of variable length chunks of the extracted text by dividing the file into discrete pieces and hashing each piece individually, wherein hash logic is based on Context Triggered Piecewise Hash (CTPH), and wherein a rolling checksum is used to produce trigger values resulting in the ordered sequence of hashes; and
determining the DLP match with one of the indexed files based on comparing the ordered sequence of hashes with corresponding ordered sequence of hashes of the indexed files.
2. The non-transitory computer-readable storage medium of claim 1 , wherein the determining the DLP match based on the comparing the ordered sequence of hashes utilizes a match score based on a number of the hashes that match and the DLP match is based on the match score being above a threshold.
3. The non-transitory computer-readable storage medium of claim 2 , wherein the threshold is user-configurable in value and configurable across a different profile of the indexed files.
4. The non-transitory computer-readable storage medium of claim 1 , wherein the steps further include
responsive to the DLP match based on the comparing the ordered sequences of hashes, performing an action based thereon.
5. The non-transitory computer-readable storage medium of claim 1 , wherein the steps further include
prior to the obtaining the file, obtaining a lookup table for a tenant associated with a user of the file, wherein the lookup table includes the ordered sequence of hashes indexed to the indexed files.
6. The non-transitory computer-readable storage medium of claim 5 , wherein the lookup table is created in an indexing tool, and wherein the indexed files cannot be recreated from data in the lookup table.
7. The non-transitory computer-readable storage medium of claim 1 , wherein the file is of a first file type, and wherein the file is determined to match one of the indexed files being a second file type, but having identical text therein.
8. The non-transitory computer-readable storage medium of claim 1 , wherein the file is determined to match one of the indexed files having similar text therein.
9. A method comprising:
obtaining a file to be checked for Data Loss Prevention (DLP);
determining a cryptographic hash of the file and comparing the cryptographic hash to corresponding cryptographic hashes of indexed files, wherein only fingerprints of the indexed files are shared to protect sensitive information;
responsive to a match between the cryptographic hash and one of the corresponding cryptographic hashes, determining a DLP match and performing an action based thereon;
responsive to no match, extracting text from the file and creating an ordered sequence of hashes of variable length chunks of the extracted text by dividing the file into discrete pieces and hashing each piece individually, wherein hash logic is based on Context Triggered Piecewise Hash (CTPH), and wherein a rolling checksum is used to produce trigger values resulting in the ordered sequence of hashes; and
determining the DLP match with one of the indexed files based on comparing the ordered sequence of hashes with corresponding ordered sequence of hashes of the indexed files.
10. The method of claim 9 , wherein the determining the DLP match based on the comparing the ordered sequence of hashes utilizes a match score based on a number of the hashes that match and the DLP match is based on the match score being above a threshold.
11. The method of claim 10 , wherein the threshold is user-configurable in value and configurable across a different profile of the indexed files.
12. The method of claim 9 , further comprising
responsive to the DLP match based on the comparing the ordered sequences of hashes, performing an action based thereon.
13. The method of claim 9 , further comprising
prior to the obtaining the file, obtaining a lookup table for a tenant associated with a user of the file, wherein the lookup table includes the ordered sequence of hashes indexed to the indexed files.
14. The method of claim 13 , wherein the lookup table is created in an indexing tool, and wherein the indexed files cannot be recreated from data in the lookup table.
15. The method of claim 9 , wherein the file is of a first file type, and wherein the file is determined to match one of the indexed files being a second file type, but having identical text therein.
16. The method of claim 9 , wherein the file is determined to match one of the indexed files having similar text therein.
17. A cloud-based system comprising:
a plurality of enforcement nodes connected to one another;
a central authority connected to the plurality of enforcement nodes; and
a Data Loss Prevention (DLP) service executed between the plurality of enforcement nodes, wherein the DLP service is configured to
obtain a file to be checked for DLP;
determine a cryptographic hash of the file and comparing the cryptographic hash to corresponding cryptographic hashes of indexed files, wherein only fingerprints of the indexed files are shared to protect sensitive information;
responsive to a match between the cryptographic hash and one of the corresponding cryptographic hashes, determine a DLP match and cause an action based thereon;
responsive to no match, extracting text from the file and creating an ordered sequence of hashes of variable length chunks of the extracted text by dividing the file into discrete pieces and hashing each piece individually, wherein hash logic is based on Context Triggered Piecewise Hash (CTPH), and wherein a rolling checksum is used to produce trigger values resulting in the ordered sequence of hashes; and
determine the DLP match with one of the indexed files based on comparing the ordered sequence of hashes with corresponding ordered sequence of hashes of the indexed files.
18. The cloud-based system of claim 17 , wherein the DLP match is determined based on a match score based on a number of the hashes that match and the DLP match is based on the match score being above a threshold.
19. The cloud-based system of claim 17 , wherein the DLP service is configured to
responsive to the DLP match based on the comparing the ordered sequences of hashes, cause an action based thereon.
20. The cloud-based system of claim 17 , wherein the DLP service is configured to
obtain a lookup table for a tenant associated with a user of the file, wherein the lookup table includes the ordered sequence of hashes indexed to the indexed files.