Systems and methods for providing faster data access using lookup and relationship tables
Disclosed are methods, systems, and computer-readable medium for providing query results. The query results may be provided more accurately and faster than via distributed tables. Disclosed techniques include receiving a plurality of tables comprising attributes and attribute data, the attribute data comprising attribute forms and attribute related information, identifying attributes and corresponding attribute data from each of the plurality of tables, applying a categorization machine learning model to categorize each attribute data as an attribute form or an attribute related information, generating the lookup table based on the attributes and the corresponding attribute forms, generating the relationship table based on the attributes and the corresponding attribute related information, receiving a query for attribute data, and generating a query response based on at least one of the lookup table or the relationship table.
1. A method for decreasing data access time using a lookup table and a relationship table, the method comprising:
receiving a trained categorization machine learning model trained using training data including a plurality of training tables comprising attribute data corresponding to attribute forms having a one-to-one relationship with an attribute among a plurality of attributes or attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes, in which the attribute forms are tagged as attribute forms and the attribute related information is tagged as attribute related information;
receiving a plurality of tables comprising a plurality of attributes and attribute data, the attribute data comprising attribute forms having a one-to-one relationship with an attribute among the plurality of attributes and attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes;
identifying attributes and corresponding attribute data from each of the plurality of tables;
applying the trained categorization machine learning model to categorize each attribute data as an attribute form having a one-to-one relationship with an attribute among the plurality of attributes or an attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes;
generating the lookup table based on the attributes and the corresponding attribute forms having a one-to-one relationship with an attribute among the plurality of attributes;
generating the relationship table based on the attributes and the corresponding attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes;
receiving a query for attribute data; and
generating a query response based on at least one of the lookup table or the relationship table.
2. The method of claim 1 , wherein the lookup table is further based on attribute related information corresponding to the attributes.
3. The method of claim 1 , wherein the plurality of tables are provided by one or more remote databases.
4. The method of claim 1 , wherein identifying attributes and corresponding attribute data from each of the plurality of tables comprises:
providing the plurality of tables to an identification machine learning model; and
receiving an output comprising the identified attributes and corresponding attribute data from the identification machine learning model.
5. The method of claim 1 , wherein the lookup table and the relationship table are a unified table.
6. The method of claim 1 , wherein at least one of the lookup table or the relationship table is labeled.
7. The method of claim 6 , wherein the label is based on one or more of a pushdown table prefix, a hash of a server index, a hash of a job identifier, a database connection index, a hash of a timestamp, a pushdown table type, or a counter.
8. The method of claim 1 , further comprising releasing at least one of the lookup table or the relationship table based on a space quota and a last use time.
9. A system for decreasing data access time using a lookup table and a relationship table, the system comprising:
at least one memory storing instructions; and
at least one processor executing the instructions to perform operations, the operations comprising:
receiving a trained categorization machine learning model trained using training data including a plurality of training tables comprising attribute data corresponding to attribute forms having a one-to-one relationship with an attribute among a plurality of attributes or attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes, in which the attribute forms are tagged as attribute forms and the attribute related information is tagged as attribute related information;
receiving a plurality of tables comprising a plurality of attributes and attribute data, the attribute data comprising attribute forms having a one-to-one relationship with an attribute among the plurality of attributes and attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes;
identifying attributes and corresponding attribute data from each of the plurality of tables;
applying the trained categorization machine learning model to categorize each attribute data as an attribute form having a one-to-one relationship with an attribute among the plurality of attributes or an attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes;
generating the lookup table based on the attributes and the corresponding attribute forms having a one-to-one relationship with an attribute among the plurality of attributes;
generating the relationship table based on the attributes and the corresponding attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes;
receiving a query for attribute data; and
generating a query response based on at least one of the lookup table or the relationship table.
10. The system of claim 9 , wherein the lookup table is further based on attribute related information corresponding to the attributes.
11. The system of claim 9 , wherein the plurality of tables are provided by one or more remote databases.
12. The system of claim 9 , wherein identifying attributes and corresponding attribute data from each of the plurality of tables comprises:
providing the plurality of tables to an identification machine learning model; and
receiving an output comprising the identified attributes and corresponding attribute data from the identification machine learning model.
13. The system of claim 9 , wherein the lookup table and the relationship table are a unified table.
14. The system of claim 9 , wherein at least one of the lookup table or the relationship table is labeled.
15. The system of claim 14 , wherein the label is based on one or more of a pushdown table prefix, a hash of a server index, a hash of a job identifier, a database connection index, a hash of a timestamp, a pushdown table type, or a counter.
16. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations, the operations comprising:
receiving a trained categorization machine learning model trained using training data including a plurality of training tables comprising attribute data corresponding to attribute forms having a one-to-one relationship with an attribute among a plurality of attributes or attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes, in which the attribute forms are tagged as attribute forms and the attribute related information is tagged as attribute related information;
receiving a plurality of tables comprising a plurality of attributes and attribute data, the attribute data comprising attribute forms having a one-to-one relationship with an attribute among the plurality of attributes and attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes;
identifying attributes and corresponding attribute data from each of the plurality of tables;
applying the trained categorization machine learning model to categorize each attribute data as an attribute form having a one-to-one relationship with an attribute among the plurality of attributes or an attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes;
generating a lookup table based on the attributes and the corresponding attribute forms having a one-to-one relationship with an attribute among the plurality of attributes;
generating a relationship table based on the attributes and the corresponding attribute related information not having a one-to-one relationship with an attribute among the plurality of attributes;
receiving a query for attribute data; and
generating a query response based on at least one of the lookup table or the relationship table.
17. The non-transitory computer-readable medium of claim 16 , wherein the lookup table is further based on attribute related information corresponding to the attributes.
18. The non-transitory computer-readable medium of claim 16 , wherein the plurality of tables are provided by one or more remote databases.
19. The non-transitory computer-readable medium of claim 16 , wherein identifying attributes and corresponding attribute data from each of the plurality of tables comprises:
providing the plurality of tables to an identification machine learning model; and
receiving an output comprising the identified attributes and corresponding attribute data from the identification machine learning model.
20. The non-transitory computer-readable medium of claim 16 , wherein the lookup table and the relationship table are a unified table.