Ascribing a confidence factor for identifying a given column in a structured dataset belonging to a particular sensitive type
Various aspects of the subject technology relate to systems, methods, and machine-readable media for determining a confidence factor for a sensitive type. The method includes applying a set of matching procedures to cells in a structured data set, the structured data set comprising columns and/or rows. The method also includes counting hit counts for the cells, the hit counts corresponding to successful matches. The method also includes counting null counts for the cells, the null counts corresponding to cells having null or invalid values. The method also includes counting mishit counts for the cells, the mishit counts corresponding to cells that are not null and do not result in a match. The method also includes calculating the confidence factor based on the hit counts, the null counts, and the mishit counts, the confidence factor providing an effective probability that cells in the structured data set is of the sensitive type.
1. A method for automated evaluation and encryption of a column of data in a structured data set, said method including:
receiving, at a computer, a structured electronic data set including at least one column of data cells, wherein said at least one column of data cells includes a header;
receiving, at said computer, a set of electronic identifiers of at least one sensitive data type;
comparing said set of electronic identifiers to said data cells of said at least one column of data cells;
electronically determining a hit count for said at least one column of data cells wherein said hit count corresponds to the number of data cells matching said at least one sensitive data type;
electronically determining a null count for said at least one column of data cells wherein said null count corresponds to the number of data cells having null or invalid values;
electronically determining a mishit count for said at least one column of data cells wherein said mishit count corresponds to the number of data cells that are not null and do not match said at least one sensitive data type;
electronically determining a confidence factor based on the hit count, the null count, and the mishit count, the wherein said confidence factor is based on the probability that any data cell in said structured data set matches said at least one sensitive data type;
when said column header matches said at least one sensitive data type, retrieving a predetermined, stored header weight from a memory;
electronically determining a weighted confidence factor based on said confidence factor and said header weight;
retrieving a predetermined, stored weighted confidence factor threshold from said memory; and
encrypting said at least one column of data cells when said weighted confidence factor is at least equal to said weighted confidence factor threshold.
2. A computerized system for automated evaluation and encryption of a column of data in a structured data set, said system including:
a memory storing a predetermined, stored header weight and a predetermined, stored weighted confidence factor threshold; and
a computerized data analysis system operating on a computer, wherein said computerized data analysis system receives:
a structured electronic data set including at least one column of data cells, wherein said at least one column of data cells includes a header; and
a set of electronic identifiers of at least one sensitive data type,
wherein said computerized data analysis system compares said set of electronic identifiers to said data cells of said at least one column of data cells and
electronically determines a hit count for said at least one column of data cells wherein said hit count corresponds to the number of data cells matching said at least one sensitive data type;
electronically determines a null count for said at least one column of data cells wherein said null count corresponds to the number of data cells having null or invalid values;
electronically determines a mishit count for said at least one column of data cells wherein said mishit count corresponds to the number of data cells that are not null and do not match said at least one sensitive data type;
electronically determines a confidence factor based on the hit count, the null count, and the mishit count, wherein said confidence factor is based on the probability that any data cell in said structured data set matches said at least one sensitive data type;
wherein, when said column header matches said at least one sensitive data type, said computerized data analysis system retrieves said predetermined, stored header weight from said memory and
electronically determines a weighted confidence factor based on said confidence factor and said header weight;
retrieves said predetermined, stored weighted confidence factor threshold from said memory; and
encrypts said at least one column of data cells when said weighted confidence factor is at least equal to said weighted confidence factor threshold.