Monitoring a stale data queue for deletion events
A computer-implemented method, system, and computer-readable media are disclosed herein. In embodiments, the computer-implemented method may entail receiving, by a data service, live data associated with an entity. The entity may be, for example, a customer of the data service. The method may then route the live data to a dual-queue system of the data service. The live data may be loaded into a live data queue of the dual queue system for processing. Processing may entail generating summary statistics from the live data. An alert may then be transmitted to the customer in response to detecting the occurrence of one or more alert events. In embodiments, the alert events may include events identified in the summary statistics. Additional embodiments are described and/or claimed.
1. A method comprising:
receiving live data at a data queue system associated with an entity, wherein the live data is associated with the entity;
routing the live data to a dual-queue within the data queue system, wherein the dual-queue includes each of a live data queue and a partition included in a persistent data store of the data queue system, wherein each of the live data queue and the partition are assigned to the entity with which the data queue system is associated and the partition serves as a stale data queue for the dual-queue; and
loading a first subset of the live data into the live data queue;
detecting that a capacity limit for the live data queue has been reached; and
responsive to detecting that a capacity limit for the live data queue has been reached, loading a second subset of the live data into the stale data queue.
2. The method of claim 1 , further comprising:
detecting one or more events associated with the live data, and
in response to detecting the one or more events associated with the live data, transmitting an alert to the entity with which the data queue system is associated, wherein the alert is based on the one or more detected events.
3. The method of claim 1 , further comprising:
detecting that an amount of data within the stale data queue exceeds a threshold;
in response to detecting that the amount of data within the stale data queue exceeds the threshold, transmitting an alert to the entity with which the data queue system is associated.
4. The method of claim 1 , further comprising instantiating the data queue system on computing resources dedicated to the entity with which the data queue system is associated.
5. The method of claim 1 further comprising transmitting at least a portion of the live data to a destination computing device and deleting the portion of the live data responsive to reception of the portion of the live data at the destination computing device.
6. The method of claim 1 , wherein the live data is included within a live data stream, the method further comprising:
segregating the live data stream into data sets based at least in part on criteria for evaluating contents of the live data stream, wherein the live data corresponds to an individual data set of the data sets; and
wherein routing the live data to the dual-queue within the data queue system comprising routing the live data to the dual-queue based on an association between the dual-queue and the individual data set.
7. The method of claim 1 , further comprising processing the live data, wherein processing the live data comprises modifying a format of the live data.
8. The method of claim 1 , further comprising:
obtaining a request for the live data; and
satisfying the request by returning a combination data from the live data queue and data from the stale data queue.
9. A system comprising:
a data store storing computer-executable instructions; and
a processor configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:
receive live data at a data queue system associated with an entity, wherein the live data is associated with the entity;
route the live data to a dual-queue within the data queue system, wherein the dual-queue includes each of a live data queue and a partition included in a persistent data store of the data queue system, wherein each of the live data queue and the partition are assigned to the entity with which the data queue system is associated and the partition serves as a stale data queue for the dual-queue; and
load a first subset of the live data into the live data queue;
detect that a capacity limit for the live data queue has been reached; and
responsive to detecting that a capacity limit for the live data queue has been reached, load a second subset of the live data into the stale data queue.
10. The system of claim 9 , wherein execution of the computer-executable instructions further causes the system to:
detect that an amount of data within the stale data queue exceeds a threshold;
in response to detecting that the amount of data within the stale data queue exceeds the threshold, transmit an alert to an entity associated with the data queue system.
11. The system of claim 9 , wherein execution of the computer-executable instructions further causes the system to instantiate the data queue system on dynamically instantiated computing resources provided by one or more servers providing the dynamically instantiated computing resources on an on-demand basis.
12. The system of claim 9 , wherein the live data comprises data regarding operation of a software application executing on a source computing device.
13. The system of claim 9 , wherein execution of the computer-executable instructions further causes the system to:
obtain a request for the live data; and
satisfy the request by returning a combination data from the live data queue and data from the stale data queue.
14. The system of claim 9 , wherein the live data is included within a live data stream, and wherein execution of the computer-executable instructions further causes the system to:
segregate the live data stream into data sets based at least in part on criteria for evaluating contents of the live data stream, wherein the live data corresponds to a first data set of the data sets; and
wherein routing the live data to the dual-queue within the data queue system comprising routing the live data to the dual-queue based on an association between the dual-queue and the first data set.
15. The system of claim 9 , wherein execution of the computer-executable instructions further causes the system to process the live data, wherein processing the live data comprises modifying a format of the live data.
16. One or more non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system, causes the computing system to:
receive live data at a data queue system associated with an entity, wherein the live data is associated with the entity;
route the live data to a dual-queue within the data queue system, wherein the dual-queue includes each of a live data queue and a partition included in a persistent data store of the data queue system, wherein each of the live data queue and the partition are assigned to the entity with which the data queue system is associated and the partition serves as a stale data queue for the dual-queue; and
load a first subset of the live data into the live data queue;
detect that a capacity limit for the live data queue has been reached; and
responsive to detecting that a capacity limit for the live data queue has been reached, load a second subset of the live data into the stale data queue.
17. The one or more non-transitory computer-readable media of claim 16 , wherein execution of the computer-executable instructions further causes the computing system to:
detect that an amount of data within the stale data queue exceeds a threshold;
in response to detecting that the amount of data within the stale data queue exceeds the threshold, transmit an alert to an entity associated with the data queue system.
18. The one or more non-transitory computer-readable media of claim 16 , wherein execution of the computer-executable instructions further causes the computing system to instantiate the data queue system on dynamically instantiated computing resources provided by one or more servers providing the dynamically instantiated computing resources on an on-demand basis.
19. The one or more non-transitory computer-readable media of claim 16 , wherein the live data comprises data regarding operation of a software application executing on a source computing device.
20. The one or more non-transitory computer-readable media of claim 16 , wherein execution of the computer-executable instructions further causes the computing system to:
obtain a request for the live data; and
satisfy the request by returning a combination data from the live data queue and data from the stale data queue.
21. The one or more non-transitory computer-readable media of claim 16 , wherein the live data is included within a live data stream, and wherein execution of the computer-executable instructions further causes the computing system to:
segregate the live data stream into data sets based at least in part on criteria for evaluating contents of the live data stream, wherein the live data corresponds to a first data set of the data sets; and
wherein routing the live data to the dual-queue within the data queue system comprising routing the live data to the dual-queue based on an association between the dual-queue and the first data set.
22. The one or more non-transitory computer-readable media of claim 16 , wherein execution of the computer-executable instructions further causes the computing system to process the live data, wherein processing the live data comprises modifying a format of the live data.