Standardizing values of a dataset

Inventors: Sean Philip Kandel (San Francisco, CA); Zain Asgar (San Francisco, CA); Wei Zheng (San Carlos, CA); Philip John Vander Broek (San Francisco, CA)

Assignee: Trifacta Inc.

G06F16/215 G06F16/221 G06F16/285 G06F2216/03

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 10,824,606
App. No.: 15/352,332
Granted: Nov 3, 2020
Kind: B1

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A system standardizes values that occur in large datasets before the dataset is analyzed. The system identifies values in a dataset that are similar to each other and associates those values with each other to form groups. The system determines a canonical value for each group of associated values. Within each group, the system replaces values that have been associated with each other with the canonical value for the group. As a result, the dataset is transformed into a dataset that has standardized values, and the standardized dataset is provided as input for analysis by a data analysis system. By standardizing the dataset in this manner, the data analysis system can process a larger portion of the dataset.

Claims (102)

1. A method for standardizing values of a dataset, comprising:

receiving a dataset for analysis by a data analysis system, the dataset comprising a plurality of records, each record having a plurality of attributes, each of the attributes having a value;

determining a sample of the dataset comprising a subset of the plurality of records;

using the sample of the dataset to perform steps comprising:

selecting at least a first record and a second record of the sample of the dataset, each comprising respective values for at least a first attribute and a second attribute;

generating a feature vector comprising at least a first distance between a value of the first attribute of the first record and a value of the first attribute of the second record and a second distance between a value of the second attribute of the first record and a value of the second attribute of the second record;

determining based on the feature vector that the first attribute is suitable for grouping records of the dataset;

identifying, from the sample, values of the first attribute of records of the subset of records;

grouping the identified values of the first attribute into one or more groups, wherein each value of a group of the one or more groups has at least a threshold similarity to values of the group; and

identifying a canonical value for each of the one or more groups, wherein each canonical value is a representative value of the attribute for the group;

standardizing values of the dataset using the groups determined using the sample of the dataset, the standardizing comprising:

identifying a value of the first attribute of a record,

identifying a group of the one or more groups corresponding to the value of the first attribute of the record, based upon a level of similarity between the value of the first attribute of the record and the canonical value of the group, and

replacing the value of the first attribute of the record with the canonical value of the group; and

providing the standardized dataset as input for analysis by a data analysis system.

2. The method of claim 1 , further comprising:

receiving one or more user inputs indicating whether the values of the first attribute of the first and second records represent the same value, and whether the values of the second attribute of the first and second records represent the same value;

determining that the first attribute is suitable for grouping records of the dataset based upon the received user inputs.

3. The method of claim 1 , wherein identifying a group corresponding to the value of the first attribute of the record comprises:

determining whether a value of the first attribute of the record is within a threshold distance from a canonical value associated with the group; and

responsive to determining that the value is within the threshold distance, including the value in the group.

4. The method of claim 1 , wherein the canonical value associated with the group represents an aggregate value based on values in the group.

5. The method of claim 1 , wherein the canonical value associated with the group is the value with the highest frequency of occurrence in the sample of the dataset.

6. The method of claim 1 , further comprising:

for a group of values, receiving a measure of strictness, the measure of strictness specifying a degree of similarity of values within the group; and

determining whether to add a value to the group based on the measure of strictness.

7. The method of claim 6 , further comprising:

determining a threshold distance value for a group based on a value of the measure of strictness for the group; and

adding a new value to the group responsive to determining that a distance between the new value and a canonical value for the group is below the threshold distance.

8. The method of claim 7 , wherein the threshold distance is a first threshold distance and the value of the measure of strictness is the first value of the measure of strictness, the method further comprising:

receiving a second value of the measure of strictness for the group, wherein the second value is indicative of higher strictness compared to the first value of the measure of strictness;

determining a second threshold distance value for a group based on the second value of measure of strictness for the group; and

adding subsequent values to the group responsive to determining that a distance between the new value and a canonical value for the group is below the second threshold distance.

9. The method of claim 1 , further comprising:

determining a measure of accuracy for performing grouping based on a column of the dataset, the column comprising the values of attributes for a plurality of records in the dataset; and

responsive to determining that the accuracy for performing grouping based on the column is lower than a threshold value, using the column for distributing records of the dataset across a plurality of processors.

10. A non-transitory computer-readable storage medium storing computer executable code for standardizing values of a dataset, the code comprising instructions for:

receiving a dataset for analysis by a data analysis system, the dataset comprising a plurality of records, each record having one or more attributes, each of the attributes having a value;