Systems and methods for tracking of data records

1 . A computer implemented method for tracking a data object in a data pipeline, the method including:

receiving, at a first component, a data object from a data source, the data object indicating an occurrence of an information technology event, wherein the first component is a collection point;

assigning, with the first component, a record number to the data object;

determining a first tracking stamp for the data object, the first tracking stamp including the record number;

broadcasting the first tracking stamp to a channel of a record tracking queue;

initiating a record in the record tracking queue for the data object, the record including the record number, a station array, a station code, and a timestamp array, wherein the first tracking stamp updates the station array, station code, and timestamp array;

transferring the data object to a second component;

determining a second tracking stamp for the data object, the second tracking stamp including the record number;

broadcasting the second tracking stamp to the channel;

updating the record with the second tracking stamp to indicate that the data object was received by the second component;

determining that the data object was not received at a downstream component, wherein the downstream component is downstream of the first component and the second component, wherein the determination is based on information for the data object in the tracking queue which indicates that the data object was transferred to the second component but was transferred to the downstream component; and

retransferring the data object from the second component to the downstream component, thereby improving operation of the data pipeline.

2 . The method of claim 1 , wherein the first tracking stamp further includes a plurality of fields, the plurality of fields including a record number field that includes the record number.

3 . The method of claim 2 , wherein the plurality of fields further includes a first record queue field, a first record time entry field, and a first counter field,

wherein the first record queue field records that the first tracking stamp was created at the collection point;

wherein the first record time entry field records the time that the data object was received by the collection point; and

wherein the first counter field records what number interconnected component the data object was received by.

4 . The method of claim 1 , wherein the record is stored as a log file in the record tracking queue.

5 . The method of claim 1 , wherein the updating the record to indicate that the data object was received by the second component further includes:

updating the station array to include a string listing the second component;

updating the timestamp array to include a time the data object was received by the second data object; and

updating the station code to two.

6 . The method of claim 1 , further including:

transferring the data object to a third component;

determining a third tracking stamp for the data object, the third tracking stamp including the record number;

broadcasting the third tracking stamp to the channel; and

updating the record to incorporate that the data object was received by the third component.

7 . The method of claim 1 , further including:

attempting to transfer the data object to a fourth component, wherein the data object does not transfer to the fourth component;

upon a set interval of time occurring, applying a scan to the record tracking queue;

finding a most recent entry of the record number in the record;

identifying a most recent entry in the station queue array of the record, the most recent entry indicating the data object was most recently transferred to the second component; and

transferring the data object to the fourth component.

8 . A computer implemented system for tracking a data object in a data pipeline, the system comprising:

a memory having processor-readable instructions stored therein; and

at least one processor configured to access the memory and execute the processor-readable instructions to perform operations including:

receiving, at a first component, a data object from a data source, the data object indicating an occurrence of an information technology event, wherein the first component is a collection point;

assigning, with the first component, a record number to the data object;

determining a first tracking stamp for the data object, the first tracking stamp including the record number;

broadcasting the first tracking stamp to a channel of a record tracking queue;

initiating a record in the record tracking queue for the data object, the record including the record number, a station array, a station code, and a timestamp array, wherein the first tracking stamp updates the station array, station code, and timestamp array;

transferring the data object to a second component;

determining a second tracking stamp for the data object, the second tracking stamp including the record number;

broadcasting the second tracking stamp to the channel;

updating the record with the second tracking stamp to indicate that the data object was received by the second component;

determining that the data object was not received at a downstream component, wherein the downstream component is downstream of the first component and the second component, wherein the determination is based on information for the data object in the tracking queue which indicates that the data object was transferred to the second component but was transferred to the downstream component; and

retransferring the data object from the second component to the downstream component, thereby improving operation of the data pipeline.

9 . The system of claim 8 , wherein the first tracking stamp further includes a plurality of fields, the plurality of fields including: a record number field that includes the record number.

10 . The system of claim 9 , wherein the plurality of fields further includes a first record queue field, a first record time entry field, and a first counter field,

wherein the first record queue field records that the first tracking stamp was created at the collection point;

wherein the first record time entry field records the time that the data object was received by the collection point; and

wherein the first counter field records what number interconnected component the data object was received by.

11 . The system of claim 8 , wherein the record is stored as a log file in the record tracking queue.

12 . The system of claim 8 , wherein the updating the record to indicate that the data object was received by the second component further includes:

updating the station array to include a string listing the second component;

updating the timestamp array to include a time the data object was received by the second data object; and

updating the station code to two.

13 . The system of claim 8 , further including:

transferring the data object to a third component;

determining a third tracking stamp for the data object, the third tracking stamp including the record number;

broadcasting the third tracking stamp to the channel; and

updating the record to incorporate that the data object was received by the third component.

14 . The system of claim 8 , further including:

attempting to transfer the data object to a fourth component, wherein the data object does not transfer to the fourth component;

upon a set interval of time occurring, applying a scan to the record tracking queue;

finding a most recent entry of the record number in the record;

identifying a most recent entry in the station queue array of the record, the most recent entry indicating the data object was most recently transferred to the second component; and

transferring the data object to the fourth component.

15 . A non-transitory computer readable medium configured to store processor-readable instructions which, when executed by at least one processor, cause the at least one processor to perform operations in a data pipeline including:

receiving, at a first component, a data object from a data source, the data object indicating an occurrence of an information technology event, wherein the first component is a collection point;

assigning, with the first component, a record number to the data object;

determining a first tracking stamp for the data object, the first tracking stamp including the record number;

broadcasting the first tracking stamp to a channel of a record tracking queue;

initiating a record in the record tracking queue for the data object, the record including the record number, a station array, a station code, and a timestamp array, wherein the first tracking stamp updates the station array, station code, and timestamp array;

transferring the data object to a second component;

determining a second tracking stamp for the data object, the second tracking stamp including the record number;

broadcasting the second tracking stamp to the channel;

updating the record with the second tracking stamp to indicate that the data object was received by the second component;

determining that the data object was not received at a downstream component, wherein the downstream component is downstream of the first component and the second component, wherein the determination is based on information for the data object in the tracking queue which indicates that the data object was transferred to the second component but was transferred to the downstream component; and

retransferring the data object from the second component to the downstream component, thereby improving operation of the data pipeline.

16 . The non-transitory computer readable medium of claim 15 , wherein the first tracking stamp further includes a plurality of fields, the plurality of fields including: a record number field that includes the record number.

17 . The non-transitory computer readable medium of claim 16 , wherein the plurality of fields further includes a first record queue field, a first record time entry field, and a first counter field,

wherein the first record queue field records that the first tracking stamp was created at the collection point;

wherein the first record time entry field records the time that the data object was received by the collection point; and

wherein the first counter field records what number interconnected component the data object was received by.

18 . The non-transitory computer readable medium of claim 15 , wherein the updating the record to indicate that the data object was received by the second component further includes:

updating the station array to include a string listing the second component;

updating the timestamp array to include a time the data object was received by the second data object; and

updating the station code to two.

19 . The non-transitory computer readable medium of claim 15 , further including:

transferring the data object to a third component;

determining a third tracking stamp for the data object, the third tracking stamp including the record number;

broadcasting the third tracking stamp to the channel; and

updating the record to incorporate that the data object was received by the third component.

20 . The non-transitory computer readable medium of claim 15 , further including:

attempting to transfer the data object to a fourth component, wherein the data object does not transfer to the fourth component;

upon a set interval of time occurring, applying a scan to the record tracking queue;

finding a most recent entry of the record number in the record;

identifying a most recent entry in the station queue array of the record, the most recent entry indicating the data object was most recently transferred to the second component; and

transferring the data object to the fourth component.