Ascribing a confidence factor for identifying a given column in a structured dataset belonging to a particular sensitive type
Various aspects of the subject technology relate to systems, methods, and machine-readable media for determining a confidence factor for a sensitive type. The method includes applying a set of matching procedures to cells in a structured data set, the structured data set comprising columns and/or rows. The method also includes counting hit counts for the cells, the hit counts corresponding to successful matches. The method also includes counting null counts for the cells, the null counts corresponding to cells having null or invalid values. The method also includes counting mishit counts for the cells, the mishit counts corresponding to cells that are not null and do not result in a match. The method also includes calculating the confidence factor based on the hit counts, the null counts, and the mishit counts, the confidence factor providing an effective probability that cells in the structured data set is of the sensitive type.
1. A method for automated evaluation and encryption of a column of data in a structured data set, said method including:
receiving, at a computer, a structured electronic data set including at least one column of data cells;
receiving, at said computer, a set of electronic identifiers of at least one sensitive data type;
comparing said set of electronic identifiers to said data cells of said at least one column of data cells;
electronically determining a hit count for said at least one column of data cells wherein said hit count corresponds to the number of data cells matching said at least one sensitive data type;
electronically determining a null count for said at least one column of data cells wherein said null count corresponds to the number of data cells having null or invalid values;
electronically determining a mishit count for said at least one column of data cells wherein said mishit count corresponds to the number of data cells that are not null and do not match said at least one sensitive data type;
electronically determining a confidence factor based on the hit count, the null count, and the mishit count, wherein said confidence factor is based on the probability that any data cell in said structured data set matches said at least one sensitive data type;
retrieving a predetermined, stored confidence factor threshold from a memory; and
encrypting said at least one column of data cells when said confidence factor is at least equal to said confidence factor threshold.
2. A system for automated evaluation and encryption of a column of data in a structured data set, said system including:
a computerized data analysis system, wherein said computerized data analysis system receives:
a structured electronic data set including at least one column of data cells; and
a set of electronic identifiers of at least one sensitive data type,
wherein said computerized data analysis system compares said set of electronic identifiers to said data cells of said at least one column of data cells and
electronically determines a hit count for said at least one column of data cells wherein said hit count corresponds to the number of data cells matching said at least one sensitive data type;
electronically determines a null count for said at least one column of data cells wherein said null count corresponds to the number of data cells having null or invalid values;
electronically determines a mishit count for said at least one column of data cells wherein said mishit count corresponds to the number of data cells that are not null and do not match said at least one sensitive data type;
electronically determines a confidence factor based on the hit count, the null count, and the mishit count, wherein said confidence factor is based on the probability that any data cell in said structured data set matches said at least one sensitive data type;
retrieves a predetermined, stored confidence factor threshold from a memory; and
encrypts said at least one column of data cells when said confidence factor is at least equal to said confidence factor threshold.