Smart result filtration for secret scanning
The disclosed techniques for secret scanning in cloud environments significantly enhance detection quality and efficiency by employing a multivariate collaborative filtering model. This model intelligently filters out irrelevant secrets identified across customer environments by analyzing a variety of features including secret type, secret value, file path, and other metadata. A relevance score is assigned to each secret based on its similarity to secrets detected in other customers. This approach intelligently filters out irrelevant secrets, reducing the noise and focusing on genuine security risks, thereby enhancing the efficiency and efficacy of secret scanning in cloud environments. The disclosed filtration process also adapts to evolving cloud environments.
1 . A method comprising:
identifying a plurality of secrets in a plurality of files, wherein the plurality of secrets comprises a plurality of instances of text and associated metadata;
determining a plurality of features of the plurality of secrets based on the associated metadata other than the plurality of instances of text and other than patterns used to identify the plurality of instances of text;
inferring, with a machine learning model, a plurality of relevance scores of the plurality of secrets based on the plurality of features;
removing, from the plurality of secrets, secrets with relevance scores that are less than a relevance threshold; and
causing at least one of the plurality of remaining secrets to be displayed.
2 . The method of claim 1 , further comprising:
ordering the plurality of selected secrets in descending order of relevance score.
3 . The method of claim 1 , further comprising:
obtaining, from the machine learning model, an explanation of a relevance score of one of the plurality of secrets; and
causing the explanation of the relevance score to be displayed.
4 . The method of claim 1 , wherein the plurality of files are stored on a cloud storage device or a relational database table.
5 . The method of claim 1 , wherein a regular expression is applied to the plurality of files to identify the plurality of secrets.
6 . The method of claim 1 , wherein the plurality of secrets comprises passwords, private keys, security tokens, certificates, or connection strings.
7 . The method of claim 1 , further comprising:
identifying a cloud resource that one of the plurality of selected secrets enables access to; and
causing an indication of the cloud resource to be displayed.
8 . A system comprising:
a processing unit; and
a computer-readable storage medium having computer-executable instructions stored thereupon, which, when executed by the processing unit, cause the processing unit to:
identify a plurality of secrets in a plurality of files, wherein the plurality of secrets comprises a plurality of instances of text and associated metadata;
determine a plurality of features of the plurality of secrets based on the associated metadata other than the plurality of instances of text and other than patterns used to identify the plurality of instances of text;
infer, with a machine learning model, a plurality of relevance scores of the plurality of secrets based on the plurality of features;
remove, from the plurality of secrets, secrets with relevance scores that are less than a relevance threshold;
infer, with the machine learning model, a remediation procedure that protects at least one of the plurality of selected secrets; and
cause the at least one of the plurality of remaining secrets and an indication of the remediation procedure to be displayed.
9 . The system of claim 8 , wherein the computer-executable instructions further cause the processing unit to:
perform the remediation procedure.
10 . The system of claim 8 , wherein the plurality of files include source code files, configuration files, text files, or documents.
11 . The system of claim 8 , wherein the plurality of files are associated with a user account, and wherein the machine learning model uses collaborative filtering to generate a relevance score of the plurality of relevance scores based on how often one of the plurality of secrets is found in other user accounts.
12 . The system of claim 11 , wherein the relevance score is based on how often the one of the plurality of secrets is found in a file with a same path in the other user accounts.
13 . The system of claim 8 , wherein the computer-executable instructions further cause the processing unit to:
determine a secret type of one of the plurality of secrets; and
select features to provide to the machine learning model for the one of the plurality of secrets based on the secret type.
14 . The system of claim 8 , wherein features for the one of the plurality of secrets are obtained from a file of the plurality of files that the one of the plurality of secrets was found in.
15 . A computer-readable storage medium having encoded thereon non-transitory computer-readable instructions that when executed by a processing unit causes a system to:
identify a plurality of secrets in a plurality of files, wherein the plurality of secrets comprises a plurality of instances of text and associated metadata;
determine a plurality of features of the plurality of secrets based on the associated metadata other than the plurality of instances of text and other than patterns used to identify the plurality of instances of text;
infer, with a machine learning model, a plurality of relevance scores of the plurality of secrets based on the plurality of features;
removing, from the plurality of secrets, secrets with relevance scores that are less than a relevance threshold; and
cause at least one of the plurality of remaining secrets to be displayed.
16 . The non-transitory computer-readable storage medium of claim 15 , wherein one of the plurality of features of includes text proximate to the one of the plurality of secrets.
17 . The non-transitory computer-readable storage medium of claim 16 , wherein the text comprises a username or an account identifier proximate to the one of the plurality of secrets.
18 . The non-transitory computer-readable storage medium of claim 15 , wherein the machine learning model is refined with training data comprising features of a secret and a training relevance score.
19 . The non-transitory computer-readable storage medium of claim 15 , wherein determining a feature of one of the plurality of secrets comprises extracting the feature from the one of the plurality of secrets.
20 . The non-transitory computer-readable storage medium of claim 15 , wherein the plurality of features comprise an offset of an individual secret within an individual file, a secret type, a file name, a file path, a file size, or a file contents.