Activating a coprocessor to transform queued data associated with a topic
Disclosed herein are methods, apparatuses and systems for transforming data within a queuing system, including activating a first coprocessor for a first topic in the queuing system based on a first script identified for the first topic, obtaining a first initialization offset identifying a position in a queue of data elements associated with the first topic, storing a first coprocessor offset based on the first initialization offset, obtaining a last offset identifying a position of a last received data element in the queue of data elements associated with the topic, executing the first coprocessor against a first data element from a queue of data elements based on a comparison between the first coprocessor offset and the last offset to produce a first transformed data element, and storing the first transformed data element in a first materialized topic associated with the first topic.
1 . A method for transforming data within a queuing system comprising:
activating a first coprocessor for a first topic in the queuing system based on a first script identified for the first topic, wherein the first topic is identified in the queuing system by a first name and the first script is associated with the first topic in a coprocessor lookup map;
obtaining a first initialization offset identifying a position in a queue of data elements associated with the first topic;
storing a first coprocessor offset based on the first initialization offset;
obtaining a last offset identifying a position of a last received data element in the queue of data elements associated with the first topic;
executing the first coprocessor against a first data element from the queue of data elements by comparing the first coprocessor offset and the last offset to produce a first transformed data element; and
storing the first transformed data element in a first materialized topic associated with the first topic, wherein the first materialized topic is identified in the queuing system by a second name different from the first name of the first topic, and wherein storage of the first data element persists at least until after the first transformed data element is stored.
2 . The method of claim 1 , further comprising:
activating a second coprocessor for the first topic in the queuing system based on a second script identified for the first topic;
obtaining a second initialization offset identifying a position in the queue of data elements associated with the first topic;
storing a second coprocessor offset based on the second initialization offset;
executing the second coprocessor against the first data element from the queue of data elements by comparing the second coprocessor offset and the last offset to produce a second transformed data element; and
storing the second transformed data element in a second materialized topic associated with the first topic.
3 . The method of claim 2 , wherein the first coprocessor and the second coprocessor have different latencies for processing data elements, and wherein the first coprocessor offset and the second coprocessor offset independently change based at least on the respective latencies for the first and second coprocessors.
4 . The method of claim 3 , wherein the first and second coprocessors execute asynchronously with respect to a receipt of data elements on which the first and second coprocessors are executed against and data elements are cached in memory pending execution of the first and second coprocessors.
5 . The method of claim 4 , wherein at least some data elements are not cached in memory if sufficient backpressure exists for the first coprocessor or the second coprocessor.
6 . The method of claim 5 , wherein a pacemaker processor provides data elements to the first coprocessor at a rate based on maintaining a desired backpressure of data elements waiting for the first coprocessor.
7 . The method of claim 2 , wherein the first and the second coprocessors are limited to an execution latency per data element less than an expected rate of receipt of data elements on which the first and second coprocessors are executed against.
8 . The method of claim 2 , wherein the first and the second coprocessors are limited to a maximum execution latency of 500 microseconds.
9 . The method of claim 1 , wherein executing the first coprocessor is performed by one or more coprocessor processes isolated from one or more queuing processes that perform at least one other step.
10 . The method of claim 9 , wherein the one or more coprocessor processes have a priority at least an order of magnitude less than the one or more queuing processes.
11 . The method of claim 1 , further comprising:
storing the first script in a compacted queue topic; and
monitoring for updates to the compacted queue topic, by a processing thread executed by a first core of a first computing device;
wherein the first script is identified for the first topic based on receipt of an update to the compacted queue topic based on the storing of the first script in the compacted queue topic.
12 . The method of claim 1 , wherein a processing thread that receives new data elements is restricted from writing to the first materialized topic.
13 . The method of claim 1 , wherein a processing thread that received and stored the data elements executes on a computing device.
14 . The method of claim 13 , wherein the processing thread operates on a first core of the computing device and utilizes a mathematically proven fault tolerant protocol,
wherein storing the first transformed data element in a first materialized topic does not utilize the mathematically proven fault tolerant protocol.
15 . The method of claim 1 , further comprising:
disabling the first coprocessor responsive to an error associated with the first coprocessor.