System and method to assure data quality in distributed data collection pipeline
The integrity of an end-to-end message sent from a source to a destination in a distributed data pipeline is improved by tracking the number of messages sent to and received by each of the intermediate components and sink components of the pipeline as the messages move through the distributed data pipeline.
1. A component of a distributed data pipeline, the component comprising:
a memory; and
a processor coupled to the memory, the processor configured to read information from and write information to the memory to:
determine a number of messages that should have been received based on a number of sent messages indicated in an incoming notification received from an upstream component during a first time period;
determine a number of incoming messages that were actually received;
determine whether the number of incoming messages that were actually received combined with an error tolerance number is less than the number of messages that should have been received;
generate a lost message signal when the number of incoming messages that were actually received combined with the error tolerance number is less than the number of messages that should have been received;
transform, upon the generation of the lost message signal, the number of incoming messages that were actually received into a number of outgoing messages based on a transformation function of the component, wherein the transformation function is predefined at the component to determine the number of messages expected to be received at a next level of the component; and
transmit an outgoing notification that indicates at least the number of outgoing messages that were transmitted as a number of outgoing messages that were sent.
2. The component of claim 1 , wherein the processor is further configured to:
receive a number of incoming messages and the incoming notification that indicates the number of incoming messages that were sent, wherein the number of incoming messages and the number of outgoing messages are different.
3. The component of claim 2 , wherein the processor is further configured to:
transmit the number of outgoing messages; and
determine a number of outgoing messages that were transmitted, wherein the outgoing notification further indicates whether the lost message signal was generated.
4. The component of claim 3 , wherein:
the number of incoming messages that were actually received is determined by counting the incoming messages as the incoming messages are received, and
the number of outgoing messages that were transmitted is determined by counting the outgoing messages as the outgoing messages are transmitted.
5. The component of claim 3 , wherein the number of incoming messages is included in a header of an incoming message.
6. The component of claim 1 , wherein the processor is further configured to:
receive a number of incoming messages;
transmit the number of outgoing messages;
determine a number of outgoing messages that were transmitted;
determine the number of messages that should have been received from the number of outgoing messages; and
receive a notification that indicates the number of outgoing messages that were actually received.
7. A method of operating a component of a distributed data pipeline, the method comprising:
determining, by a processor, a number of messages that should have been received based on a number of sent messages indicated in an incoming notification received from an upstream component during a first time period;
determining, by the processor, a number of incoming messages that were actually received;
determining, by the processor, whether the number of incoming messages that were actually received combined with an error tolerance number is less than the number of messages that should have been received;
generating, by the processor, a lost message signal when the number of incoming messages that were actually received combined with the error tolerance number is less than the number of messages that should have been received;
transforming, by the processor upon the generation of the lost message signal, the number of incoming messages that were actually received into a number of outgoing messages based on a transformation function of the component, wherein the transformation function is predefined at the component to determine the number of messages expected to be received at a next level of the component; and
transmitting, by the processor, an outgoing notification that indicates at least the number of outgoing messages that were transmitted as a number of outgoing messages that were sent.
8. The method of claim 7 , further comprising:
receiving, by the processor, a number of incoming messages and the incoming notification that indicates the number of incoming messages that were sent, wherein the number of incoming messages and the number of outgoing messages are different.
9. The method of claim 8 , further comprising:
transmitting, by the processor, the number of outgoing messages; and
determining, by the processor, a number of outgoing messages that were transmitted, wherein the outgoing notification further indicates whether the lost message signal was generated.
10. The method of claim 9 , wherein:
the number of incoming messages that were actually received is determined by counting the incoming messages as the incoming messages are received, and
the number of outgoing messages that were transmitted is determined by counting the outgoing messages as the outgoing messages are transmitted.
11. The method of claim 9 , wherein the number of incoming messages is included in a header of an incoming message.
12. The method of claim 7 , further comprising:
receiving, by the processor, a number of incoming messages;
transmitting, by the processor, the number of outgoing messages;
determining, by the processor, a number of outgoing messages that were transmitted, and the number of outgoing messages determining the number of messages that should have been received; and
receiving, by the processor, a notification that indicates the number of outgoing messages that were actually received.
13. A non-transitory computer-readable storage medium, having embedded therein program instructions, which when executed by a processor cause the processor to execute a method of operating a component in a distributed data pipeline, the method comprising:
determining a number of messages that should have been received based on a number of sent messages indicated in an incoming notification received from an upstream component during a first time period;
determining a number of incoming messages that were actually received;
determining whether the number of incoming messages that were actually received combined with an error tolerance number is less than the number of messages that should have been received;
generating a lost message signal when the number of messages that were actually received combined with the error tolerance number is less than the number of messages that should have been received;
transforming, upon the generation of the lost message signal, the number of incoming messages that were actually received into a number of outgoing messages based on a transformation function of the component, wherein the transformation function is predefined at the component to determine the number of messages expected to be received at a next level of the component; and
transmitting an outgoing notification that indicates at least the number of outgoing messages that were transmitted as a number of outgoing messages that were sent.
14. The medium of claim 13 , wherein the method further comprises:
receiving a number of incoming messages and the incoming notification that indicates the number of incoming messages that were sent, wherein the number of incoming messages and the number of outgoing messages are different.
15. The medium of claim 14 , wherein the method further comprises:
transmitting the number of outgoing messages; and
determining a number of outgoing messages that were transmitted, wherein the outgoing notification further indicates whether the lost message signal was generated.
16. The medium of claim 15 , wherein:
the number of incoming messages that were actually received is determined by counting the incoming messages as the incoming messages are received, and
the number of outgoing messages that were transmitted is determined by counting the outgoing messages as the outgoing messages are transmitted.
17. The medium of claim 15 , wherein the number of incoming messages is included in a header of an incoming message.
18. The medium of claim 13 , wherein the method further comprises:
receiving a number of incoming messages, wherein the number of incoming messages and the number of outgoing messages are different;
transmitting the number of outgoing messages;
determining a number of outgoing messages that were transmitted, the number of outgoing messages determining the number of messages that should have been received; and
receiving a notification that indicates the number of outgoing messages that were actually received.
19. A distributed data pipeline, comprising:
a source component configured to:
generate a number of messages;
transmit the number of messages;
determine a number of messages that were transmitted;
transmit a notification that indicates a number of messages that should have been received from the number of messages that were transmitted; and
a first-level component coupled to the source component, the first level component configured to:
receive the number of messages;
determine a number of messages that should have been received from the notification;
determine a number of incoming messages that were actually received;
determine whether the number of incoming messages that were actually received combined with an error tolerance number is less than the number of messages that should have been received;
generate a lost message signal when the number of incoming messages that were actually received combined with the error tolerance number is less than the number of messages that should have been received; and
transform, upon the generation of the lost message signal, the incoming messages that were actually received into a number of outgoing messages based on a transformation function of the first-level component, wherein the transformation function is predefined at the first-level component to determine the number of messages expected to be received at a next level of the first-level component, wherein the number of incoming messages that were actually received and the number of outgoing messages are different.
20. The distributed data pipeline of claim 19 , wherein the first-level component is further configured to:
transmit the number of outgoing messages;
determine a number of outgoing messages that were transmitted; and
transmit an outgoing notification that indicates the number of outgoing messages that were transmitted as a number of outgoing messages that were sent, and further indicates whether the lost message signal was generated.