Data analysis system and method
This disclosure relates to a system and method for data analysis. According to a first aspect, there is described a method, the method being performed using one or more processors, comprising: receiving one or more user inputs indicative of one or more relationships between data in a plurality of datasets; determining, based on the one or more user inputs, at least one object view for visualizing the data in the plurality of datasets; generating, based on the one or more user inputs, metadata comprising: an object graph indicative of the one or more relationships between two or more of the plurality of datasets; and information identifying the at least one object view; and in response to a query relating to the plurality of datasets, using the metadata to determine how response data responding to the query should be provided.
1. A method, the method being performed using one or more processors, comprising:
accessing a plurality of datasets, wherein the plurality of datasets includes tables such that each table comprises rows representing objects and columns representing properties;
accessing statistics relating to the plurality of datasets;
comparing the statistics with one or more thresholds corresponding to types of global properties that are present in the plurality of datasets, wherein each global property corresponds to one or more columns that are associated with a first portion of data in two or more of the plurality of datasets;
based at least in part on the comparing, performing a first determination that at least one of the one or more thresholds has been met;
based on the first determination, inferring that a first global property is present; and
generating metadata comprising the first global property.
2. The method of claim 1 , further comprising:
further based on the determination, generating instructions configured to display, by a user interface, the first global property as a suggestion to a user.
3. The method of claim 2 , further comprising:
receiving, by the user interface, indication of acceptance of the first global property.
4. The method of claim 3 , further comprising:
linking at least two datasets based on the datasets having rows or columns of data where the data meets a threshold related to the first global property.
5. The method of claim 1 , further comprising:
based at least in part on the inferring, automatically linking at least two datasets based on the datasets having rows or columns of data where the data meets a threshold related to the first global property.
6. The method of claim 1 , wherein the statistics are received from a remote server.
7. The method of claim 1 , wherein the statistics are determined by using the one or more processors.
8. The method of claim 1 , wherein the statistics comprise one or more of: a proportion of datasets comprising an object type or property; a uniqueness score; cardinalities of object types or properties; and a title key.
9. The method of claim 1 , wherein the one or more thresholds comprise one or more of: a minimum proportion of datasets containing an object type or property; a minimum uniqueness score; and a maximum or minimum cardinality of object types or properties.
10. The method of claim 1 , wherein the metadata further comprises a join between at least two datasets of the plurality of datasets.
11. The method of claim 1 , wherein the one or more thresholds comprises at least one of: a proportion of datasets comprising an object type or property; a uniqueness score; cardinalities of object types and/or properties; or a title key.
12. The method of claim 1 , further comprising:
based on the first global property, inferring one or more joins between datasets of the plurality of datasets.
13. The method of claim 1 , wherein the first global property is defined by one or more data formats or one or more data types.
14. The method of claim 1 , further comprising:
synchronizing the metadata with a metadata layer stored on a server.
15. A system comprising:
a computer readable storage medium having program instructions embodied therewith; and one or more processors configured to execute the program instructions to cause the system to perform the method of claim 1 .
16. A computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform the method of claim 1 .
17. The method of claim 1 , further comprising:
based on the first determination, linking or joining the at least two or more of the plurality of datasets, wherein the linking or joining is based on (1) a user selection or (2) automatically performed in response to the inference that the first global property is present.
18. The method of claim 1 , wherein the first global property is further determined or inferred based on identification or detection of a first data format or a first data type corresponding to the first portion of data.
19. The method of claim 1 , wherein the metadata is configured to be used to determine how to generate response data corresponding to a user query pertaining to the plurality of datasets, and wherein the metadata indicates (1) one or more relationships between the plurality of datasets, and (2) default display settings for how to display data corresponding to the plurality of datasets.
20. The method of claim 19 , wherein the default display settings include settings for display prominence or an object view, wherein the object view comprises one or more visualizations that include one or more of a histogram, a bar chart, a pie chart, a list, a dashboard view, and a line graph.