Data movement and monitoring system
Disclosed herein are various embodiments for a data movement and monitoring system. A variety of embodiments operate by determining a move time and a run time associated with a job configured to move a set of data from a table of a first system to one or more tables of a second system. A job deadline is calculated, and it is determined that a job is delayed based on the job deadline being prior to the current time and an absence of the set of data from a status table that indicates which data has been moved from the first system to the second system. A record in the status table is updated to indicate the job is delayed, and a notification that the job is delayed is provided to the one or more accounts.
1 . A computer-implemented method, comprising:
determining, from a schedule, a move time and a run time associated with a job configured to move a set of data from a table of a first system to one or more tables of a second system, the move time indicating when the job is scheduled to be executed, and the run time indicating a length of time to execute the job;
calculating a job deadline corresponding to the move time plus the run time, wherein the job deadline for the job is prior to a current time;
determining that the job is delayed;
identifying a computing process that utilizes data from the one or more tables of the second system, wherein one or more accounts are associated with the identified computing process; and
providing a notification that the job is delayed to the one or more accounts associated with the identified computing process.
2 . The method of claim 1 , wherein the first system comprises a local computing environment and wherein the second system comprises a cloud computing environment configured to load the set of data into one or more cloud hosted tables.
3 . The method of claim 1 , further comprising:
determining that the set of data was moved from the first system to the second system after the job deadline but before the current time, wherein the identified computing process was scheduled to execute after the job deadline but before the current time.
4 . The method of claim 1 , wherein the determining that the job is delayed comprises:
updating a record in a status table to indicate the job is delayed, wherein the status table indicates which data has been moved from the first system to the second system.
5 . The method of claim 4 , further comprising:
identifying a production time associated with the set of data associated with the job, the production time indicating when the set of data was produced; and
determining that the production time of the set of data associated with the job was after the move time, wherein when the production time is after the move time, an extension time is applied before the record in the status table is updated as delayed.
6 . The method of claim 5 , further comprising:
calculating the extension time comprising a difference between the production time and the move time; and
calculating the job deadline corresponding to the move time plus the run time plus the extension time.
7 . The computer-implemented method of claim 1 , further comprising:
providing a second notification that the job is delayed to another computing process configured to execute based on the one or more tables of the second system.
8 . A system, comprising:
a memory; and
at least one processor coupled to the memory and configured to perform operations comprising:
determining, from a schedule, a move time and a run time associated with a job configured to move a set of data from a table of a first system to one or more tables of a second system, the move time indicating when the job is scheduled to be executed, and the run time indicating a length of time to execute the job;
calculating a job deadline corresponding to the move time plus the run time, wherein the job deadline for the job is prior to a current time;
determining that the job is delayed;
identifying a computing process that utilizes data from the one or more tables of the second system, wherein one or more accounts are associated with the identified computing process; and
providing a notification that the job is delayed to the one or more accounts associated with the identified computing process.
9 . The system of claim 8 , wherein the first system comprises a local computing environment and wherein the second system comprises a cloud computing environment configured to load the set of data into one or more cloud hosted tables.
10 . The system of claim 8 , the operations further comprising:
determining that the set of data was moved from the first system to the second system after the job deadline but before the current time, wherein the identified computing process was scheduled to execute after the job deadline but before the current time.
11 . The system of claim 8 , wherein the determining that the job is delayed comprises:
updating a record in a status table to indicate the job is delayed, wherein the status table indicates which data has been moved from the first system to the second system.
12 . The system of claim 11 , the operations further comprising:
identifying a production time associated with the set of data associated with the job, the production time indicating when the set of data was produced; and
determining that the production time of the set of data associated with the job was after the move time, wherein when the production time is after the move time, an extension time is applied before the record in the status table is updated as delayed.
13 . The system of claim 12 , the operations further comprising:
calculating the extension time comprising a difference between the production time and the move time; and
calculating the job deadline corresponding to the move time plus the run time plus the extension time.
14 . The system of claim 8 , the operations further comprising:
providing a second notification that the job is delayed to another computing process configured to execute based on the one or more tables of the second system.
15 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:
determining, from a schedule, a move time and a run time associated with a job configured to move a set of data from a table of a first system to one or more tables of a second system, the move time indicating when the job is scheduled to be executed, and the run time indicating a length of time to execute the job;
calculating a job deadline corresponding to the move time plus the run time, wherein the job deadline for the job is prior to a current time;
determining that the job is delayed;
identifying a computing process that utilizes data from the one or more tables of the second system, wherein one or more accounts are associated with the identified computing process; and
providing a notification that the job is delayed to the one or more accounts associated with the identified computing process.
16 . The non-transitory computer-readable medium of claim 15 , wherein the first system comprises a local computing environment and wherein the second system comprises a cloud computing environment configured to load the set of data into one or more cloud hosted tables.
17 . The non-transitory computer-readable medium of claim 15 , the operations further comprising:
determining that the set of data was moved from the first system to the second system after the job deadline but before the current time, wherein the identified computing process was scheduled to execute after the job deadline but before the current time.
18 . The non-transitory computer-readable medium of claim 15 , wherein the determining that the job is delayed comprises:
updating a record in a status table to indicate the job is delayed, wherein the status table indicates which data has been moved from the first system to the second system.
19 . The non-transitory computer-readable medium of claim 18 , the operations further comprising:
identifying a production time associated with the set of data associated with the job, the production time indicating when the set of data was produced; and
determining that the production time of the set of data associated with the job was after the move time, wherein when the production time is after the move time, an extension time is applied before the record in the status table is updated as delayed.
20 . The non-transitory computer-readable medium of claim 15 , the operations further comprising:
providing a second notification that the job is delayed to another computing process configured to execute based on the one or more tables of the second system.