Correlating messages from asynchronous message queues using polling with exponential backoff and jitter
There are provided systems and methods for correlating messages from asynchronous message queues using polling with exponential backoff and jitter. A service provider, such as an electronic transaction processor for digital transactions, may provide computing services to users, which may be used to engage in interactions with other users and entities including for electronic transaction processing. When utilizing these transaction processing services, merchant locations may utilize payment terminals, such as card and NFC readers, as well as point-of-sale devices, each of which may independently send transaction messages to the service provider. To correlate these messages from different queues, the messages may be stored when received in an intermediary storage independently and the other storage polled at intervals with exponential backoff to identify corresponding messages and data. Further, jitter may be introduced to the polling process to stagger polling events for different messages.
1 . A system comprising:
a non-transitory memory; and
one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising:
receiving, from a first message queue, a first transaction message associated with a first transaction processing component of the system, wherein the first transaction message originates from a card transaction at a point of sale (POS) terminal;
storing the first transaction message in a database tagged with a first domain identifier associated with the first transaction processing component;
selecting the first domain identifier as a primary domain identifier for message polling;
polling the database at designated intervals for a second transaction message associated with the first transaction message using message data associated with the first transaction message and a second domain identifier, wherein the second transaction message further originates from the card transaction and is stored in a second message queue and is associated with a second transaction processing component corresponding to the second domain identifier;
identifying the second transaction message after at least one of the designated intervals based on the polling;
correlating the first transaction message and the second transaction message usable for one or more transaction processing operations of the card transaction;
combining the first transaction message and the second transaction message for data processing by a computing service;
generating a combined data container based on the combined first and second transaction message, wherein the combined data container comprises actionable data for the data processing by the computing service; and
deleting the first transaction message and the second transaction message from the database subsequent to the combining.
2 . The system of claim 1 , wherein the first message queue comprises one of a card payment message queue or POS message queue and the first transaction processing component comprises one of the POS terminal or a payment instrument reader at the POS terminal, and wherein the second message queue comprises another one of the card payment message queue or the POS message queue and the second transaction processing component comprises another one of the POS terminal or the payment instrument reader.
3 . The system of claim 2 , wherein the storing the first transaction message comprises storing a plurality of transaction messages including the first transaction message and the second transaction message each with a corresponding one of a plurality of domain identifiers at different times based on times of receipt of the plurality of transaction messages.
4 . The system of claim 1 , wherein the first transaction message is stored to an ephemeral queue storage in a cloud computing system that is cleared of the first transaction message and the second transaction message after the correlating.
5 . The system of claim 1 , wherein the polling is performed using an exponential backoff to the designated intervals that increases a time period between consecutive ones of the designated intervals.
6 . The system of claim 5 , wherein the operations further comprise:
introducing a jitter parameter to the polling performed using the exponential backoff.
7 . The system of claim 6 , wherein introducing the jitter parameter prevents or limits synchronous polling of the database for different correlations of a plurality of transaction messages.
8 . The system of claim 1 , wherein the operations further comprise:
generating, based on the combined data container, a transaction data object associated with the first transaction message and the second transaction message; and
processing the transaction data object using a transaction data processing stack of the system.
9 . A method comprising:
receiving, from a first message queue, a first transaction message associated with a first transaction processing component of a service provider system, wherein the first transaction message originates from a card transaction at a point of sale (POS) terminal;
storing the first transaction message in a database tagged with a first domain identifier associated with the first transaction processing component;
selecting the first domain identifier as a primary domain identifier for message polling;
polling the database at designated intervals for a second transaction message associated with the first transaction message using message data associated with the first transaction message and a second domain identifier, wherein the second transaction message further originates from the card transaction and is stored in a second message queue and is associated with a second transaction processing component corresponding to the second domain identifier;
identifying the second transaction message after at least one of the designated intervals based on the polling;
correlating the first transaction message and the second transaction message usable for one or more transaction processing operations of the card transaction;
combining the first transaction message and the second transaction message for data processing by a computing service;
generating a combined data container based on the combined first and second transaction message, wherein the combined data container comprises actionable data for the data processing by the computing service; and
deleting the first transaction message and the second transaction message from the database subsequent to the combining.
10 . The method of claim 9 , wherein the first message queue comprises one of a card payment message queue or POS message queue and the first transaction processing component comprises one of the POS terminal or a payment instrument reader at the POS terminal, and wherein the second message queue comprises another one of the card payment message queue or the POS message queue and the second transaction processing component comprises another one of the POS terminal or the payment instrument reader.
11 . The method of claim 10 , wherein the storing the first transaction message comprises storing a plurality of transaction messages including the first transaction message and the second transaction message each with a corresponding one of a plurality of domain identifiers at different times based on times of receipt of the plurality of transaction messages.
12 . The method of claim 9 , wherein the first transaction message is stored to an ephemeral queue storage in a cloud computing system that is cleared of the first transaction message and the second transaction message after the correlating.
13 . The method of claim 9 , wherein the polling is performed using an exponential backoff to the designated intervals that increases a time period between consecutive ones of the designated intervals.
14 . The method of claim 13 , further comprising:
introducing a jitter parameter to the polling performed using the exponential backoff.
15 . The method of claim 14 , wherein introducing the jitter parameter prevents or limits synchronous polling of the database for different correlations of a plurality of transaction messages.
16 . The method of claim 9 , further comprising:
generating, based on the combined data container, a transaction data object associated with the first transaction message and the second transaction message; and
processing the transaction data object using a transaction data processing stack of the service provider system.
17 . A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:
receiving, from a first message queue, a first transaction message associated with a first transaction processing component of a service provider system, wherein the first transaction message originates from a card transaction at a point of sale (POS) terminal;
storing the first transaction message in a database tagged with a first domain identifier associated with the first transaction processing component;
selecting the first domain identifier as a primary domain identifier for message polling;
polling the database at designated intervals for a second transaction message associated with the first transaction message using message data associated with the first transaction message and a second domain identifier, wherein the second transaction message further originates from the card transaction and is stored in a second message queue and is associated with a second transaction processing component corresponding to the second domain identifier;
identifying the second transaction message after at least one of the designated intervals based on the polling;
correlating the first transaction message and the second transaction message usable for one or more transaction processing operations of the card transaction;
combining the first transaction message and the second transaction message for data processing by a computing service;
generating a combined data container based on the combined first and second transaction message, wherein the combined data container comprises actionable data for the data processing by the computing service; and
deleting the first transaction message and the second transaction message from the database subsequent to the combining.
18 . The non-transitory machine-readable medium of claim 17 , wherein the first message queue comprises one of a card payment message queue or POS message queue and the first transaction processing component comprises one of the POS terminal or a payment instrument reader at the POS terminal, and wherein the second message queue comprises another one of the card payment message queue or the POS message queue and the second transaction processing component comprises another one of the POS terminal or the payment instrument reader.
19 . The non-transitory machine-readable medium of claim 18 , wherein the storing the first transaction message comprises storing a plurality of transaction messages including the first transaction message and the second transaction message each with a corresponding one of a plurality of domain identifiers at different times based on times of receipt of the plurality of transaction messages.
20 . The non-transitory machine-readable medium of claim 17 , wherein the first transaction message is stored to an ephemeral queue storage in a cloud computing system that is cleared of the first transaction message and the second transaction message after the correlating.