Granted Patent B2

US 10,942,947 · App. 15/900,289 · Granted Mar 9, 2021

Inventors: Caitlin Colgrove (Palo Alto, CA); Harsh Pandey (New York, NY); Gabrielle Javitt (New York, NY)

Assignee: Palantir Technologies Inc.

G06F16/284G06F16/23G06F16/2456

Abstract

Claims (72)

1. A system comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the system to perform:

identifying a first dataset from one or more databases and a second dataset from the one or more databases, the first dataset having first data, and the second dataset having second data;

condensing the first data into first condensed data, the condensing the first data including applying a hash function to the first dataset, the first condensed data comprising first hash values generated from the applying the hash function to the first dataset;

condensing the second data into second condensed data, the condensing the second data including applying the hash function to the second dataset, the second condensed data comprising second hash values generated from the applying the hash function to the second dataset;

determining proportions of unique values in each of columns in the first dataset and the second dataset based on the first hash values and the second hash values;

estimating a degree of overlap between a column of the columns in the first dataset and a column of the columns of the second dataset based on the first hash values and the second hash values, and based on the proportions of entries having unique values in each of the columns in the first dataset and the second dataset;

determining parameters for any one or more of a left join, an inner join, an outer join, and a right join operation, the parameters comprising:

numbers of rows to keep, match, or add from the first dataset; and

numbers of rows to keep, match, or add from the second dataset,

the determined parameters being based at least in part on the proportions of entries having unique values; and

suggesting a join operation, from any one or more of the left join, the inner join, the outer join, and the right join, based on any of:

the proportions of unique values in each of the columns; and

the estimated degree of overlap;

computing a first relationship measure for the first dataset, the first relationship measure including the first condensed data;

computing a second relationship measure for the second dataset, the second relationship measure including the second condensed data;

computing a join key using the first relationship measure and the second relationship measure, and using the determined parameters, the join key representing a correspondence area between the first dataset and the second dataset; and

selecting an operation, from any one or more of the left join, the inner join, the outer join, and the right join, based on the join key.

2. The system of claim 1 , wherein the join operation is suggested based on an overlap suggestion measure based on the first relationship measure and the second relationship measure.

3. The system of claim 2 , wherein the overlap suggestion measure comprises a null measure to identify a null portion of the first dataset or the second dataset.

4. The system of claim 2 , wherein the overlap suggestion measure comprises one or more of: a first uniqueness measure configured to identify a first unique portion of the first dataset, and a second uniqueness measure configured to identify a second unique portion.

5. The system of claim 2 , wherein the instructions cause the system to perform configuring an interactive user interface element to display the overlap suggestion measure.

6. The system of claim 1 , wherein the first relationship measure is based on a first hash value of the first data in the first dataset.

7. The system of claim 1 , wherein the second relationship measure is based on a second hash value of the second data in the second dataset.

8. The system of claim 1 , wherein the correspondence area comprises one or more of:

a left correspondence area configured to represent the first dataset and left matching data from the second dataset, the left matching data matching at least a portion of the first dataset;

a right correspondence area configured to represent the second dataset and right matching data from the first dataset, the right matching data matching at least a portion of the second dataset;

an inner correspondence area configured to represent inner matching data representing only an overlapping portion of the first dataset and the second dataset; and

an outer correspondence area configured to represent outer matching data representing the first dataset and the second dataset.

9. The system of claim 1 , wherein the first dataset comprises a first column of a first database of the one or more databases.

10. The system of claim 9 , wherein the second dataset comprises a second column of a second database of the one or more databases.

11. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:

identifying a first dataset from one or more databases and a second dataset from the one or more databases, the first dataset having first data, and the second dataset having second data;

condensing the first data into first condensed data, the condensing the first data including applying a hash function to the first dataset, the first condensed data comprising first hash values generated from the applying the hash function to the first dataset;

condensing the second data into second condensed data, the condensing the second data including applying the hash function to the second dataset, the second condensed data comprising second hash values generated from the applying the hash function to the second dataset;

determining proportions of unique values in each of columns in the first dataset and the second dataset based on the first hash values and the second hash values;

estimating a degree of overlap between a column of the columns in the first dataset and a column of the columns of the second dataset based on the first hash values and the second hash values, and based on the proportions of entries having unique values in each of the columns in the first dataset and the second dataset;

determining parameters for any one or more of a left join, an inner join, an outer join, and a right join operation, the parameters comprising:

numbers of rows to keep, match, or add from the first dataset; and

numbers of rows to keep, match, or add from the second dataset,

the determined parameters being based at least in part on the proportions of entries having unique values; and

suggesting a join operation, from any one or more of the left join, the inner join, the outer join, and the right join, based on any of:

the proportions of unique values in each of the columns; and

the estimated degree of overlap;

computing a first relationship measure for the first dataset, the first relationship measure including the first condensed data;

computing a second relationship measure for the second dataset, the second relationship measure including the second condensed data;

computing a join key using the first relationship measure and the second relationship measure, and using the determined parameters, the join key representing a correspondence area between the first dataset and the second dataset; and

selecting an operation, from any one or more of the left join, the inner join, the outer join, and the right join, based on the join key.

12. The method of claim 11 , wherein the join operation is suggested based on an overlap suggestion measure based on the first relationship measure and the second relationship measure.

13. The method of claim 12 , wherein the overlap suggestion measure comprises a null measure to identify a null portion of the first dataset or the second dataset.

14. The method of claim 12 , wherein the overlap suggestion measure comprises one or more of: a first uniqueness measure configured to identify a first unique portion of the first dataset, and a second uniqueness measure configured to identify a second unique portion.

15. The method of claim 12 , further comprising configuring an interactive user interface element to display the overlap suggestion measure.

16. The method of claim 11 , wherein the correspondence area comprises one or more of:

a left correspondence area configured to represent the first dataset and left matching data from the second dataset, the left matching data matching at least a portion of the first dataset;

a right correspondence area configured to represent the second dataset and right matching data from the first dataset, the right matching data matching at least a portion of the second dataset;

an inner correspondence area configured to represent inner matching data representing only an overlapping portion of the first dataset and the second dataset; and

an outer correspondence area configured to represent outer matching data representing the first dataset and the second dataset.

17. The method of claim 11 , wherein the first dataset comprises a first column of a first database of the one or more databases.

18. The system of claim 1 , wherein the instructions further cause the system to perform:

computing a null measure of the first dataset and the second dataset based on the first hash values and the second hash values; and wherein the suggesting the join operation is based on the computed null measure, and wherein the determining, for the left join, the inner join, the outer join, and the right join operation, the numbers of rows to keep, match, or add from the first dataset and from the second dataset, is further based on the null measure of the first dataset and the second dataset.

19. The system of claim 1 , wherein the instructions further cause the system to perform:

presenting:

for the left join operation, a number of rows to keep from the first dataset and a number of rows to match from the second dataset;

for the inner join operation, a number of rows to keep from the first dataset and from the second dataset;

for the outer join operation, a number of rows to keep from the first dataset and a number of rows to add from the second dataset; and

for the right join operation, a number of rows to keep from the second dataset and a number of rows to match from the first dataset.

20. The system of claim 1 , wherein the instructions further cause the system to perform:

presenting, along with a graphical depiction:

a number of rows of the first dataset having a match with corresponding rows of the second dataset;

a number of rows of the first dataset not matching any rows of the second dataset; and

a number of rows of the second dataset not matching any rows of the first dataset.

Assignments (8)

Continuity (2)