Systems and methods for distributed architecture of market-data processing device

1. A market-data-processing device (MDPD) comprising:

a line-rate processing module (LPRM); and

a host, wherein:

the LRPM is communicatively connected to an LRPM external-communication interface having (i) a first LRPM external-communication port configured to receive a market-data input feed from an upstream device and (ii) a second LRPM external-communication port configured to transmit a market-data output feed to a downstream device, the LRPM comprising a programmable logic circuit (PLC) configured to:

generate the market-data output feed based on the market-data input feed; and

transmit an archival copy of the market-data input feed to the host via a communication bus; and

the host (i) is communicatively connected to (a) the LRPM via the communication bus and (b) a host external-communication interface and (ii) comprises a host processor that is configured to:

cache the archival copy of the market-data input feed in host data storage; and

use the cached archival copy of the market-data input feed to provide, to the downstream device via the host external-communication interface, a gap-fill service for the market-data output feed.

2. The MDPD of claim 1 , further comprising:

a first circuit board on which the PLC resides; and

a second circuit board on which the host processor resides,

wherein the communication bus communicatively connects the first and second circuit boards.

3. The MDPD of claim 2 , further comprising:

a memory chip residing on the first circuit board, wherein:

the PLC is further configured to store respective associations on the memory chip between order numbers and ticker symbols from respective order-book updates received in the market-data input feed; and

the PLC being configured to generate the market-data output feed comprises the PLC being configured to:

use respective order numbers from respective later-received order-book updates in the market-data input feed to retrieve from the memory chip the respective ticker symbols from the respective stored associations; and

insert the respective retrieved ticker symbols into corresponding messages in the market-data output feed.

4. The MDPD of claim 2 , wherein the communication bus comprises a Peripheral Component Interconnect (PCI) Express (PCIe) bus.

5. The MDPD of claim 2 , the LRPM further comprising a crosspoint switch residing on the first circuit board, the crosspoint switch being configured to:

pass the market-data input feed from the first LRPM external-communication port to the PLC; and

pass the market-data output feed from the PLC to the second LRPM external-communication port.

6. The MDPD of claim 5 , wherein:

the LRPM external-communication interface further comprises a third LRPM external-communication port; and

the crosspoint switch is further configured to:

generate a duplicate copy of the market-data output feed; and

pass the duplicate copy of the market-data output feed to the third LRPM external-communication port in parallel with passing the market-data output feed to the second LRPM external-communication port.

7. The MDPD of claim 2 , wherein:

the market-data input feed comprises a plurality of order-book updates to respective ticker symbols in a plurality of ticker symbols;

the PLC is further configured to maintain an operational output-feed profile that specifies a ticker-symbol subset that identifies one or more ticker symbols from among the plurality of ticker symbols; and

the PLC being configured to generate the market-data output feed based on the market-data input feed comprises the PLC being configured to filter the market-data input feed down to a filtered market-data feed of the order-book updates to respective ticker symbols in the ticker-symbol subset.

8. The MDPD of claim 7 , wherein:

a management copy of the output-feed profile is stored in host data storage; and

the operational output-feed profile is based on the management copy of the output-feed profile.

9. The MDPD of claim 8 , wherein:

the host comprises a host user interface configured to receive a revision input for the management copy of the output-feed profile;

the host processor is further configured to, responsive to receiving the revision input:

update the management copy of the output-feed profile based on the revision input;

generate revision instructions for the operational output-feed profile based on the revision input; and

provide, via the communication bus, the revision instructions to the PLC;

the PLC is further configured to update the operational output-feed profile based on the revision instructions; and

the PLC being configured to generate the market-data output feed comprises the PLC being configured to generate the market-data output feed based on the updated operational output-feed profile.

10. The MDPD of claim 7 , wherein:

the market-data output feed comprises output-feed packets that each includes one or more order-book updates, each of which is an order-book update for a respective ticker symbol;

the PLC being configured to generate the market-data output feed comprises the PLC being configured to insert, into the output-feed packets, output-feed sequence numbers for the respective order-book updates included in the output-feed packets; and

the host processor being configured to use the cached archival copy of the market-data input feed to provide the gap-fill service for the market-data output feed to the downstream device comprises the host processor being configured to:

store, in the host data storage, input-feed-to-output-feed mapping data that correlates respective cached copies of order-book updates from the cached archival copy of the market-data input feed with the output-feed sequence numbers of corresponding order-book updates from the output-feed packets; and

retrieve, and provide to the downstream device, portions of the cached archival copy of the market-data input feed based on the corresponding output-feed sequence numbers in the stored input-feed-to-output-feed mapping data.

11. The MDPD of claim 10 , wherein the output-feed sequence numbers comprise one or more of feed-level sequence numbers, symbol-level sequence numbers, and partition-level sequence numbers.

12. The MDPD of claim 11 , wherein the host processor is further configured to use a management copy of the output-feed profile stored in host data storage to, independently of the LRPM, determine the output-feed sequence numbers for storage in the input-feed-to-output-feed mapping data.

13. The MDPD of claim 11 , wherein:

the PLC is further configured to provide the output-feed sequence numbers to the host processor via the communication bus; and

the host processor is further configured to receive, from the PLC via the communication bus, the output-feed sequence numbers for storage in the input-feed-to-output-feed mapping data.

14. The MDPD of claim 11 , wherein:

the PLC is further configured to:

generate sequence-number-record data indicative of which order-book updates from the market-data input feed had corresponding order-book updates in the generated market-data output feed; and

provide the generated sequence-number-record data to the host processor via the communication bus; and

the host processer is further configured to:

receive the generated sequence-number-record data from the PLC via the communication bus; and

use the received sequence-number-record data to determine the output-feed sequence numbers for storage in the input-feed-to-output-feed mapping data.

15. The MDPD of claim 1 , wherein the host processer is further configured to use the cached archival copy of the market-data input feed to provide, to the downstream device via the host external-communication interface, a late-join service for the market-data output feed.

16. The MDPD of claim 1 , wherein the host processer is further configured to:

maintain a current local list of active orders for each ticker symbol in the plurality of ticker symbols based on the archival copy of the market-data input feed; and

responsive to receiving, from the downstream device, a late-join request for a particular ticker symbol:

generate a late-join response from the current local list of active orders for the particular ticker symbol; and

transmit the late-join response via the host external-communication interface to the downstream device.

17. A method comprising:

receiving a market-data input feed on a first line-rate processing module (LRPM) external-communication port;

generating, by a programmable logic circuit (PLC) of an LRPM, a market-data output feed based on the market-data input feed;

transmitting the generated market-data output feed to a downstream device via a second LRPM external-communication port;

transmitting an archival copy of the market-data input feed from the LRPM to a host processer via a communication bus;

the host processor caching the archival copy of the market-data input feed in host data storage; and

the host processor using the cached archival copy of the market-data input feed to provide a gap-fill service for the market-data output feed to the downstream device via a host external-communication interface.

18. A market-data-processing device (MDPD) comprising:

a first external data port configured to receive, from an upstream device, a market-data input feed of order-book updates to respective ticker symbols in a plurality of ticker symbols;

a programmable logic circuit (PLC) configured to receive the market-data input feed and generate both (i) a market-data output feed and (ii) an archival copy of the market-data input feed;

a second external data port configured to transmit, to a downstream device, the market-data output feed; and

a host communicatively coupled to (i) the PLC via a communication bus and (ii) the downstream device via a network interface card (NIC), the host configured to:

receive and cache the archival copy of the market-data input feed; and

responsive to receiving a gap-fill request from the downstream device:

generate a gap-fill response to the gap-fill request from the cached archival copy of the market-data input feed; and

transmit the gap-fill response via the NIC to the downstream device.

19. The MDPD of claim 18 , further comprising:

a first circuit board on which the PLC resides; and

a second circuit board on which the host resides,

wherein the communication bus communicatively connects the first and second circuit boards.

20. The MDPD of claim 19 , further comprising a crosspoint switch residing on the first circuit board, the crosspoint switch being configured to:

pass the market-data input feed from the first external data port to the PLC; and

pass the market-data output feed from the PLC to the second external data port.

21. The MDPD of claim 20 , further comprising a third external data port, wherein the crosspoint switch is further configured to:

generate a duplicate copy of the market-data output feed; and

pass the duplicate copy of the market-data output feed to the third external data port in parallel with passing the market-data output feed to the second external communication port.

22. The MDPD of claim 21 , wherein:

the PLC is further configured to maintain an operational output-feed profile that specifies a ticker-symbol subset that identifies one or more ticker symbols from among the plurality of ticker symbols; and

the PLC being configured to generate the market-data output feed based on the market-data input feed comprises the PLC being configured to filter the market-data input feed down to a filtered market-data feed of the order-book updates to respective ticker symbols in the ticker-symbol subset.

23. The MDPD of claim 18 , wherein:

the market-data output feed comprises output-feed packets that each includes one or more order-book updates, each of which is an order-book update for a respective ticker symbol;

the PLC being configured to generate the market-data output feed comprises the PLC being configured to insert, into the output-feed packets, output-feed sequence numbers for the respective order-book updates included in the output-feed packets;

the host is further configured to store input-feed-to-output-feed mapping data that correlates respective cached copies of order-book updates from the cached archival copy of the market-data input feed with the output-feed sequence numbers of corresponding order-book updates from the output-feed packets; and

the host being configured to generate the gap-fill response to the gap-fill request from the cached archival copy of the market-data input feed comprises the host being configured to:

identify, in the gap-fill request, one or more requested output-feed sequence numbers;

retrieve, using the stored input-feed-to-output-feed mapping data, the one or more cached copies of order-book updates from the cached archival copy of the market-data input feed that correspond with the identified one or more requested output-feed sequence numbers from the gap-fill request; and

include in the generated gap-fill response the retrieved one or more cached copies of order-book updates from the cached archival copy of the market-data input feed.

24. The MDPD of claim 23 , wherein the output-feed sequence numbers comprise one or more of feed-level sequence numbers, symbol-level sequence numbers, and partition-level sequence numbers.

25. The MDPD of claim 24 , wherein the host is further configured to use a management copy of the output-feed profile stored in host data storage to, independently of the PLC, determine the output-feed sequence numbers for storage in the input-feed-to-output-feed mapping data.

26. The MDPD of claim 24 , wherein:

the PLC is further configured to provide the output-feed sequence numbers to the host via the communication bus; and

the host is further configured to receive, from the PLC via the communication bus, the output-feed sequence numbers for storage in the input-feed-to-output-feed mapping data.

27. The MDPD of claim 24 , wherein:

the PLC is further configured to:

generate sequence-number-record data indicative of which order-book updates from the market-data input feed had corresponding order-book updates in the generated market-data output feed; and

provide the generated sequence-number-record data to the host via the communication bus; and

the host is further configured to:

receive the generated sequence-number-record data from the PLC via the communication bus; and

use the received sequence-number-record data to determine the output-feed sequence numbers for storage in the input-feed-to-output-feed mapping data.

28. The MDPD of claim 18 , wherein the host is further configured to:

maintain a current local list of active orders for each ticker symbol in the plurality of ticker symbols based on the archival copy of the market-data input feed; and

responsive to receiving, from the downstream device, a late-join request for a particular ticker symbol:

generate a late-join response from the current local list of active orders for the particular ticker symbol; and

transmit the late-join response, via the NIC, to the downstream device.