Data transfer and resource management system
Disclosed herein are system, method, and computer program product embodiments for a data transfer and resource management system. An embodiment operates by retrieving both data and a schema from a table. A first offset corresponding to the retrieved data indicating a retrieval time of the retrieved data and a number of records is identified. It is determined that the retrieved data corresponds to the schema. The data is stored on a messaging platform and made available to one or more end users prior to a transfer to one or more cloud servers. A second offset corresponding to a number of records of the data retrieved by the one or more end users from the messaging platform is determined. It is validate that the second offset corresponds to the first offset. The records and the schema retrieved from the intermediary messaging platform are transferred to the one or more cloud servers.
1. A computer-implemented method comprising:
retrieving data from a table of a database;
retrieving a schema of the table;
identifying a first offset corresponding to the retrieved data indicating a retrieval time of the retrieved data and a number of records corresponding to the retrieved data;
determining that the retrieved data corresponds to the schema;
storing the retrieved data on an intermediary messaging platform, wherein the stored data and the schema are available to one or more end users prior to a transfer to one or more cloud servers;
determining a second offset corresponding to a number of records of the data retrieved by the one or more end users from the intermediary messaging platform;
validating that the second offset corresponds to the first offset;
transferring the records and the schema retrieved from the intermediary messaging platform to the one or more cloud servers for different data sets retrieved from the table of the database;
determining that at least one of the transferred records of one of the data sets includes an error;
based on the retrieval time of the first offset, identifying which data set corresponds to the error; and
re-retrieving the identified data set identified based on the first offset, wherein at least one of the different data sets is not re-retrieved.
2. The method of claim 1 , wherein the schema includes a table name, column names of a plurality of columns of the table, and data types for the plurality of columns of the table.
3. The method of claim 1 , further comprising:
repeating the retrieving, identifying, determining that the retrieved data corresponds to the schema, storing, determining the second offset, validating, and transferring a plurality of times for different sets of data retrieved from the table of the database and transferred to the one more cloud servers.
4. The method of claim 1 , further comprising:
identifying a plurality of sensitive data values retrieved from the table; and
encrypting the sensitive data values,
wherein storing the retrieved data comprises storing the encrypted sensitive values on the intermediary messaging platform.
5. The method of claim 1 , further comprising storing both the first offset and the second offset in a log.
6. The method of claim 1 , further comprising:
determining that the data is retrieved from the table on a regular time interval;
determining a report that is associated with the data retrieved from the table;
monitoring the retrieval of the data from the table during the regular time interval;
determining that there is no new data to retrieve from the table during a particular time interval; and
generating an empty data set for the report based on the determining that there is no new data.
7. A system comprising:
a memory; and
at least one processor coupled to the memory and configured to perform operations comprising:
retrieving data from a table of a database;
retrieving a schema of the table;
identifying a first offset corresponding to the retrieved data indicating a retrieval time of the retrieved data and a number of records corresponding to the retrieved data;
determining that the retrieved data corresponds to the schema;
storing the retrieved data on an intermediary messaging platform, wherein the stored data and the schema are available to one or more end users prior to a transfer to one or more cloud servers;
determining a second offset corresponding to a number of records of the data retrieved by the one or more end users from the intermediary messaging platform;
validating that the second offset corresponds to the first offset;
transferring the records and the schema retrieved from the intermediary messaging platform to the one or more cloud servers for different data sets retrieved from the table of the database;
determining that at least one of the transferred records of one of the data sets includes an error;
based on the retrieval time of the first offset, identifying which data set corresponds to the error; and
re-retrieving the identified data set identified based on the first offset, wherein at least one of the different data sets is not re-retrieved.
8. The system of claim 7 , wherein the schema includes a table name, column names of a plurality of columns of the table, and data types for the plurality of columns of the table.
9. The system of claim 7 , the operations further comprising:
repeating the retrieving, identifying, determining that the retrieved data corresponds to the schema, storing, determining the second offset, validating, and transferring a plurality of times for different sets of data retrieved from the table of the database and transferred to the one more cloud servers.
10. The system of claim 7 , the operations further comprising:
identifying a plurality of sensitive data values retrieved from the table; and
encrypting the sensitive data values,
wherein the operations comprising storing the retrieved data further comprise storing the encrypted sensitive values on the intermediary messaging platform.
11. The system of claim 7 , the operations further comprising storing the first offset and the second offset in a log.
12. The system of claim 7 , the operations further comprising:
determining that the data is retrieved from the table on a regular time interval;
determining a report that is associated with the data retrieved from the table;
monitoring the retrieval of the data from the table during the regular time interval;
determining that there is no new data to retrieve from the table during a particular time interval; and
generating an empty data set for the report based on the determining that there is no new data.
13. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:
retrieving data from a table of a database;
retrieving a schema of the table;
identifying a first offset corresponding to the retrieved data indicating a retrieval time of the retrieved data and a number of records corresponding to the retrieved data;
determining that the retrieved data corresponds to the schema;
storing the retrieved data on an intermediary messaging platform wherein the stored data and the schema are available to one or more end users prior to a transfer to one or more cloud servers;
determining a second offset corresponding to a number of records of the data retrieved by the one or more end users from the intermediary messaging platform;
validating that the second offset corresponds to the first offset;
transferring the records and the schema retrieved from the intermediary messaging platform to the one or more cloud servers for different data sets retrieved from the table of the database;
determining that at least one of the transferred records of one of the data sets includes an error;
based on the retrieval time of the first offset, identifying which data set corresponds to the error; and
re-retrieving the identified data set identified based on the first offset, wherein at least one of the different data sets is not re-retrieved.
14. The device of claim 13 , wherein the schema includes a table name, column names of a plurality of columns of the table, and data types for the plurality of columns of the table.
15. The device of claim 13 , the operations further comprising:
repeating the retrieving, identifying, determining that the retrieved data corresponds to the schema, storing, determining the second offset, validating, and transferring a plurality of times for different sets of data retrieved from the table of the database and transferred to the one more cloud servers.
16. The device of claim 13 , the operations further comprising:
identifying a plurality of sensitive data values retrieved from the table; and
encrypting the sensitive data values,
wherein the operations comprising storing the retrieved data further comprise storing the encrypted sensitive values on the intermediary messaging platform.
17. The device of claim 13 , the operations further comprising storing both the first offset and the second offset in a log.