System and method for generating a digital backup of database applications to allow database operations
A system and method for generating a quarriable backup of database applications to allow database operations is provided. The method includes accessing a plurality of database applications deployed in a cloud environment; exporting data stored in the database applications to a plurality of files, each file utilizing a column-oriented storage format; generating metadata based on the plurality of exported files, where the metadata includes at least a hash map of related-columns and a list of related-tables; and generating a quarriable backup based on a plurality of exported files and the metadata.
1 . A method for generating a quarriable backup of database applications to allow database operations, comprising:
accessing a plurality of database applications deployed in a cloud environment;
exporting data stored in the database applications to a plurality of files, each file utilizing a column-oriented storage format;
generating metadata based on the plurality of exported files, wherein the metadata includes at least a hash map of related-columns and a list of related-tables;
wherein generating the metadata further comprises:
normalizing column values of tables in the database applications;
wherein normalizing the column values of tables in the database applications further comprises:
determining a category for each column of each table in the database applications;
normalizing column values of each column;
wherein normalizing the column values of each column further comprises applying a normalization function selected based on the category of a respective column; and
saving the normalized column values in the quarriable backup;
computing similar values;
unifying columns names of tables in the database applications;
identifying related-columns based on at least the normalizing columns values;
determining related-tables based on the identified related-columns; and
generating a quarriable backup based on a plurality of exported files and the metadata.
2 . The method of claim 1 , wherein the plurality of exported files is in a column-oriented storage format.
3 . The method of claim 1 , wherein identifying the related-columns further comprises: computing hash-based signatures for at least two columns in the normalized column values; and designating columns as having similar values upon determining that columns share a threshold overlap in their computed hash-based signatures, wherein at least two columns having similar values are related-columns.
4 . The method of claim 3 , wherein determining the related-tables further comprises:
designating at least two tables having related-columns as related-tables.
5 . The method of claim 1 , wherein determining the category of each column further comprises: prompting a large language model (LLM) with at least a set of values of a column to determine the category of the column.
6 . The method of claim 1 , wherein the quarriable backup includes tables or data of different database applications.
7 . The method of claim 6 , further comprising:
performing the database operations on the quarriable backup including tables or data of different database applications.
8 . The method of claim 7 , wherein a database operation includes any one of: retrieve, insert, delete, and update.
9 . The method of claim 1 , wherein a database application of the database applications includes any one of: a database hosted on a virtual machine and a relational database service (RDS) running on an infrastructure of the cloud environment.
10 . A non-transitory computer-readable medium storing a set of instructions for generating a quarriable backup of database applications to allow database operations, the set of instructions comprising: one or more instructions that, when executed by one or more processors of a device, cause the device to:
access a plurality of database applications deployed in a cloud environment;
export data stored in the database applications to a plurality of files, each file utilizing a column-oriented storage format;
generate metadata based on the plurality of exported files, wherein the metadata includes at least a hash map of related-columns and a list of related-tables;
wherein generating the metadata further comprises:
normalizing column values of tables in the database applications;
wherein normalizing the column values of tables in the database applications further comprises:
determining a category for each column of each table in the database applications;
normalizing column values of each column;
wherein normalizing the column values of each column further comprises applying a normalization function selected based on the category of a respective column; and
saving the normalized column values in the quarriable backup;
computing similar values;
unifying columns names of tables in the database applications;
identifying related-columns based on at least the normalizing columns values;
determining related-tables based on the identified related-columns; and
generate a quarriable backup based on a plurality of exported files and the metadata.
11 . A system for generating a quarriable backup of database applications to allow database operations comprising: one or more processors configured to:
access a plurality of database applications deployed in a cloud environment;
export data stored in the database applications to a plurality of files, each file utilizing a column-oriented storage format;
generate metadata based on the plurality of exported files, wherein the metadata includes at least a hash map of related-columns and a list of related-tables;
wherein generating the metadata further comprises:
normalizing column values of tables in the database applications;
wherein normalizing the column values of tables in the database applications further comprises:
determining a category for each column of each table in the database applications;
normalizing column values of each column;
wherein normalizing the column values of each column further comprises applying a normalization function selected based on the category of a respective column; and
saving the normalized column values in the quarriable backup;
computing similar values;
unifying columns names of tables in the database applications;
identifying related-columns based on at least the normalizing columns values;
determining related-tables based on the identified related-columns; and
generate a quarriable backup based on a plurality of exported files and the metadata.
12 . The system of claim 11 , wherein the plurality of exported files is in a column-oriented storage format.
13 . The system of claim 11 , wherein the one or more processors, when determining the category of each column, are configured to: prompt a large language model (LLM) with at least a set of values of a column to determine the category of the column.
14 . The system of claim 11 , wherein the one or more processors, when identifying the related-columns, are configured to: compute hash-based signatures for at least two columns in the normalized column values; and designate columns as having similar values upon determining that columns share a threshold overlap in their computed hash-based signatures, wherein at least two columns having similar values are related-columns.
15 . The system of claim 13 , wherein the one or more processors, when determining the related-tables, are configured to:
designate at least two tables having related-columns as related-tables.
16 . The system of claim 11 , wherein the one or more processors, when determining the category of each column, are configured to: prompt a large language model (LLM) with at least a set of values of a column to determine the category of the column.
17 . The system of claim 11 , wherein the quarriable backup includes tables or data of different database applications.
18 . The system of claim 16 , wherein the one or more processors are further configured to:
perform the database operations on the quarriable backup including tables or data of different database applications.
19 . The system of claim 17 , wherein a database operation includes any one of:
retrieve, insert, delete, and update.
20 . The system of claim 11 , wherein a database application of the database applications includes any one of:
a database hosted on a virtual machine and a relational database service (RDS) running on an infrastructure of the cloud environment.