Rate limiting of cloud account change events and state management
Rate limiting of cloud account change events and state management is described herein. One embodiment includes instructions to process each of a first stream of change events received from a cloud provider and associated with any assets of a particular public cloud account, determine that the first stream of change events exceeds a rate threshold, discard each of a second stream of change events received from the public cloud provider and associated with any assets of the particular public cloud account, query the cloud provider to perform a collection on all the assets of the particular public cloud account after a particular delay period, and process each of a third stream of change events received from the cloud provider and associated with any assets of the particular public cloud account responsive to a completion of the collection.
1. A non-transitory machine-readable medium having instructions stored thereon which, when executed by a processor, cause the processor to:
process each of a first stream of change events received from a cloud provider and associated with any assets of a particular public cloud account;
determine that the first stream of change events exceeds a rate threshold;
discard each of a second stream of change events received from the public cloud provider and associated with any assets of the particular public cloud account;
query the cloud provider to perform a collection on all the assets of the particular public cloud account after a particular delay period; and
process each of a third stream of change events received from the cloud provider and associated with any assets of the particular public cloud account responsive to a completion of the collection.
2. The medium of claim 1 , wherein each change event is configured to trigger collection for a particular asset associated with the change event.
3. The medium of claim 1 , including instructions to perform a check for a flag associated with the particular public cloud account indicating that the first stream of change events exceeds the rate threshold.
4. The medium of claim 3 , including instructions to discard each of the second stream of change events received from the public cloud provider and associated with any assets of the particular public cloud account responsive to determining a presence of the flag.
5. The medium of claim 3 , including instructions to determine, for each change event of the first stream of change events, whether that change event causes the first stream of change events to exceed the rate threshold responsive to determining a lack of the flag.
6. The medium of claim 5 , including instructions to process a change event of the first stream of change events responsive to a determination that the change event does not cause the first stream of change events to exceed the rate threshold.
7. The medium of claim 5 , including instructions to associate the flag with the particular public cloud account indicating that the first stream of change events exceeds the rate threshold responsive to a determination that a change event of the first stream of change events causes the first stream of change events to exceed the rate threshold.
8. The medium of claim 1 , wherein the first stream of change events includes a plurality of event types, and wherein the instructions include instructions to:
determine that a particular one of the plurality of event types included in the first stream of change events exceeds the rate threshold;
discard each of a second stream of change events of the particular event type received from the public cloud provider and associated with the particular public cloud account;
query the cloud provider to perform the collection on all the assets of the particular public cloud account after the particular delay period; and
process each of the third stream of change events received from the cloud provider and associated with any assets of the particular public cloud account responsive to the completion of the collection, wherein the third stream of change events includes the plurality of event types.
9. The medium of claim 1 , wherein the instructions include instructions to:
process each of a first stream of change events received from the cloud provider and associated with any asset types of the particular public cloud account;
determine that change events corresponding to a particular asset type of the public cloud account included in the first stream of change events exceeds the rate threshold;
discard each of a second stream of change events of the particular asset type received from the public cloud provider and associated with the particular public cloud account;
query the cloud provider to perform the collection on all the assets of the particular public cloud account after the particular delay period; and
process each of a third stream of change events received from the cloud provider and associated with any asset types of the particular public cloud account responsive to a completion of the collection, wherein the third stream of change events includes the plurality of event types.
10. The medium of claim 1 , wherein the instructions include instructions to:
determine that change events corresponding to a particular asset of the public cloud account included in the first stream of change events exceeds the rate threshold;
discard each of a second stream of change events corresponding to the particular asset received from the public cloud provider and associated with the particular public cloud account;
query the cloud provider to perform the collection on all the assets of the particular public cloud account after the particular delay period; and
process each of the third stream of change events received from the cloud provider and associated with any assets of the particular public cloud account responsive to the completion of the collection.
11. A method, comprising:
processing each of a first stream of change events received from a cloud provider and associated with any assets of a particular public cloud account, wherein the first stream of change events includes a plurality of event types;
determining that a particular one of the plurality of event types included in the first stream of change events exceeds a rate threshold;
discarding each of a second stream of change events of the particular event type received from the public cloud provider and associated with the particular public cloud account;
querying the cloud provider to perform a collection on all the assets of the particular public cloud account after a particular delay period; and
processing each of a third stream of change events received from the cloud provider and associated with any assets of the particular public cloud account responsive to a completion of the collection, wherein the third stream of change events includes the plurality of event types.
12. The method of claim 11 , wherein the method includes determining the particular delay period based on an expected reduction in a rate of received change events.
13. The method of claim 11 , wherein the collection on all the assets of the particular public cloud account includes a collection of a current state of all the assets of the particular public cloud account.
14. The method of claim 11 , wherein the method includes updating a graph model associated with the particular public cloud account subsequent to the performance of the collection.
15. A system, comprising:
a first stream engine configured to process each of a first stream of change events received from a cloud provider and associated with any assets of a particular public cloud account;
a threshold engine configured to determine that the first stream of change events exceeds a rate threshold;
a discard engine configured to discard each of a second stream of change events received from the public cloud provider and associated with any assets of the particular public cloud account;
a collection engine configured to query the cloud provider to perform a collection on all the assets of the particular public cloud account after a particular delay period; and
a third stream engine configured to process each of a third stream of change events received from the cloud provider and associated with any assets of the particular public cloud account responsive to a completion of the collection.
16. The system of claim 15 , wherein the threshold engine is configured to perform a check for a flag associated with the particular public cloud account indicating that the first stream of change events exceeds the rate threshold.
17. The system of claim 16 , wherein the discard engine is configured to discard each of the second stream of change events received from the public cloud provider and associated with any assets of the particular public cloud account responsive to determining a presence of the flag.
18. The system of claim 16 , wherein the threshold engine is configured to determine, for each change event of the first stream of change events, whether that change event causes the first stream of change events to exceed the rate threshold responsive to determining a lack of the flag.
19. The system of claim 18 , wherein the first stream engine is configured to process a change event of the first stream of change events responsive to a determination that the change event does not cause the first stream of change events to exceed the rate threshold.
20. The system of claim 18 , wherein the threshold engine is configured to associate the flag with the particular public cloud account indicating that the first stream of change events exceeds the rate threshold responsive to a determination that a change event of the first stream of change events causes the first stream of change events to exceed the rate threshold.