Event-driven monitoring of resources in a cloud computing environment
Techniques for event driven harvesting and analysis of cloud computing resources in a cloud computing environment, comprising: obtaining, from a cloud computing environment, data related to an event that occurred in the cloud computing environment; in response to obtaining the data, requesting, from the cloud computing environment, supplemental data about the event that occurred in the cloud computing environment, the supplemental data including information about the event and/or information about impact of the event on a resource; determining whether a security action is to be taken at least in part by analyzing the data and/or the supplemental data; and when it is determined a security action is to be taken, performing the security action.
1. A method for event driven harvesting of data in one or more cloud computing environments, the method comprising:
using at least one computer hardware processor to perform:
obtaining, from a first cloud computing environment, first data related to a first event that occurred in the first cloud computing environment, the first event relating to a first resource in the first cloud computing environment, the first data including a first identifier for the first event;
in response to obtaining the first data,
requesting, from an application programming interface (API) of the first cloud computing environment and using the first identifier, first supplemental data about the first event, the first supplemental data including supplemental information about the first event and/or information about impact of the first event on the first resource;
determining whether a security action is to be taken at least in part by analyzing the first data and/or the first supplemental data; and
when it is determined a security action is to be taken, performing the security action.
2. The method of claim 1 , further comprising:
obtaining, from a second cloud computing environment, different from the first cloud computing environment, second data related to a second event that occurred in the second cloud computing environment, the second event relating to a second resource in the second cloud computing environment, the second data including a second identifier for the second event;
in response to obtaining the second data,
requesting, from an application programming interface (API) of the second cloud computing environment and using the second identifier, second supplemental data about the second event, the second supplemental data including supplemental information about the second event and/or information about impact of the second event on the second resource;
determining whether a second security action is to be taken at least in by analyzing the second data and/or the second supplemental data; and
when it is determined a second security action is to be taken, performing the security action.
3. The method of claim 2 , further comprising:
after obtaining the first data and the first supplemental data,
generating a first data structure; and
storing the first data and the first supplemental data in the first data structure; and
after obtaining the second data and the second supplemental data,
generating a second data structure; and
storing the second data and the second supplemental data in the second data structure,
wherein the first data structure and the second data structure are a same type of data structure.
4. The method of claim 1 , further comprising:
in response to obtaining the first data, determining whether the first supplemental data is to be requested; and
in response to determining the first supplemental data is to be requested:
generating a first data structure having a field indicating the first supplemental data is to be requested;
storing the first data in the first data structure; and
requesting the first supplemental data.
5. The method of claim 4 , further comprising:
before the requesting, storing a job associated with the first data structure in a queue, the queue storing jobs to be performed including scheduled data collections; and
prioritizing the job associated with the first data structure within the queue to be addressed before the scheduled data collections, in response to determining the first supplemental data is to be requested.
6. The method of claim 1 , wherein information about the impact of the first event on the first resource comprises information about changes to an amount of storage associated with the first resource and information about changes to metadata associated with the first resource, and wherein information about changes to metadata associated with the first resource comprises information about changes to a structure of data within the first resource, changes to software installed on the first resource, changes to permissions for accessing the first resource, and/or changes to security rules relating to the first resource.
7. The method of claim 1 , wherein performing the security action comprises performing at least one of: updating software installed in the first resource, changing a network configuration of the first resource, changing a configuration of one or more software applications executing on the first resource, changing a configuration of an operating system executing on the first resource, changing one or more permissions for the first resource, deleting malware from the first resource, removing corrupted files or data from the first resource, taking the first resource offline, killing an instance of the first resource, and/or blocking communications to and/or from the first resource.
8. The method of claim 1 , wherein analyzing the first data and the first supplemental data comprises identifying presence of one or more software bugs, one or more out-of-date software applications, one or more unpatched software applications, corrupted data, unencrypted data, one or more improper access permissions for the first resource, one or more misconfigurations, one or more computer viruses, and/or malware within the first data and the first supplemental data.
9. The method of claim 1 , wherein the first data is obtained from a queue managed within the first cloud computing environment, and the first data is pushed to the queue from a log of the first cloud computing environment.
10. The method of claim 1 , wherein the first data is obtained from an event subscription, and the requesting comprises requesting the first supplemental data from a resource API associated with the first cloud computing environment by querying the resource API for information associated with the first resource and collecting information related to the impact of the first event on the first resource.
11. The method of claim 1 , wherein the first data and/or the supplemental data includes an event type of the first event; and
further comprising:
in response to determining, based on the event type, the first event is associated with allocation of storage to the first resource, taking a snapshot of storage attached to the first resource.
12. The method of claim 11 , wherein the first cloud computing environment is an AWS cloud computing environment, and taking the snapshot is performed in response to determining the first event is an Attach Volume event type.
13. A system for event-driven harvesting of data in one or more cloud computing environments, the system comprising:
at least one computer hardware processor; and
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method comprising:
obtaining, from a first cloud computing environment, first data related to a first event that occurred in the first cloud computing environment, the first event relating to a first resource in the first cloud computing environment, the first data including a first identifier for the first event;
in response to obtaining the first data,
requesting, from an application programming interface (API) of the first cloud computing environment and using the first identifier, first supplemental data about the first event, the first supplemental data including supplemental information about the first event and/or information about impact of the first event on the first resource;
determining whether a security action is to be taken at least in part by analyzing the first data and/or the first supplemental data; and
when it is determined a security action is to be taken, performing the security action.
14. The system of claim 13 , wherein the method further comprises:
obtaining, from a second cloud computing environment, different from the first cloud computing environment, second data related to a second event that occurred in the second cloud computing environment, the second event relating to a second resource in the second cloud computing environment, the second data including a second identifier for the second event;
in response to obtaining the second data,
requesting, from an application programming interface (API) of the second cloud computing environment and using the second identifier, second supplemental data about the second event, the second supplemental data including supplemental information about the second event and/or information about impact of the second event on the second resource;
determining whether a second security action is to be taken at least in by analyzing the second data and/or the second supplemental data; and
when it is determined a second security action is to be taken, performing the security action.
15. The system of claim 14 , wherein the method further comprises:
after obtaining the first data and the first supplemental data,
generating a first data structure; and
storing the first data and the first supplemental data in the first data structure; and
after obtaining the second data and the second supplemental data,
generating a second data structure; and
storing the second data and the second supplemental data in the second data structure,
wherein the first data structure and the second data structure are a same type of data structure.
16. The system of claim 13 , wherein information about the impact of the first event on the first resource comprises information about changes to an amount of storage associated with the first resource and information about changes to metadata associated with the first resource, and wherein information about changes to metadata associated with the first resource comprises information about changes to a structure of data within the first resource, changes to software installed on the first resource, changes to permissions for accessing the first resource, and/or changes to security rules relating to the first resource.
17. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method comprising:
obtaining, from a first cloud computing environment, first data related to a first event that occurred in the first cloud computing environment, the first event relating to a first resource in the first cloud computing environment, the first data including a first identifier for the first event;
in response to obtaining the first data,
requesting, from an application programming interface (API) of the first cloud computing environment and using the first identifier, first supplemental data about the first event, the first supplemental data including supplemental information about the first event and/or information about impact of the first event on the first resource;
determining whether a security action is to be taken at least in part by analyzing the first data and/or the first supplemental data; and
when it is determined a security action is to be taken, performing the security action.
18. The at least one non-transitory computer-readable storage medium of claim 17 , wherein the method further comprises:
obtaining, from a second cloud computing environment, different from the first cloud computing environment, second data related to a second event that occurred in the second cloud computing environment, the second event relating to a second resource in the second cloud computing environment, the second data including a second identifier for the second event;
in response to obtaining the second data,
requesting, from an application programming interface (API) of the second cloud computing environment and using the second identifier, second supplemental data about the second event, the second supplemental data including supplemental information about the second event and/or information about impact of the second event on the second resource;
determining whether a second security action is to be taken at least in by analyzing the second data and/or the second supplemental data; and
when it is determined a second security action is to be taken, performing the security action.
19. The at least one non-transitory computer-readable storage medium of claim 18 , wherein the method further comprises:
after obtaining the first data and the first supplemental data,
generating a first data structure; and
storing the first data and the first supplemental data in the first data structure; and
after obtaining the second data and the second supplemental data,
generating a second data structure; and
storing the second data and the second supplemental data in the second data structure,
wherein the first data structure and the second data structure are a same type of data structure.
20. The at least one non-transitory computer-readable storage medium of claim 17 , wherein information about the impact of the first event on the first resource comprises information about changes to an amount of storage associated with the first resource and information about changes to metadata associated with the first resource, and wherein information about changes to metadata associated with the first resource comprises information about changes to a structure of data within the first resource, changes to software installed on the first resource, changes to permissions for accessing the first resource, and/or changes to security rules relating to the first resource.