Universal cloud data ingestion for stream processing
An example method includes accessing, by a data platform via a network, data from one or more cloud environments; identifying, by the data platform and in the data, first data associated with a first entity and a first data type and second data associated with a second entity and a second data type; mapping, by the data platform and based on the first entity and the first data type, the first data to a first data stream of a data streaming platform; mapping, by the data platform and based on the second entity and the second data type, the second data to a second data stream of the data streaming platform, the second data stream different from the first data stream of the data streaming platform; and generating, based on the first data stream, a graph representing activity associated with the first entity in the one or more cloud environments.
1. A method comprising:
accessing, by a data platform via a network, data from one or more cloud environments;
identifying, by the data platform and in the data, first data associated with a first entity and a first data type and second data associated with a second entity and a second data type;
mapping, by the data platform and based on the first entity and the first data type, the first data to a first data stream of a data streaming platform; mapping, by the data platform and based on the second entity and the second data type, the second data to a second data stream of the data streaming platform, the second data stream separate from the first data stream of the data streaming platform; and
generating, based on the first data stream, a graph representing activity associated with the first entity in the one or more cloud environments.
2. The method of claim 1 , wherein a data type identifies a data source among the one or more cloud environments.
3. The method of claim 1 , wherein a data type identifies an event type associated with the data from the one or more cloud environments.
4. The method of claim 1 , further comprising:
identifying, by the data platform and in the first data stream, first subset data having a first event type and second subset data having a second event type;
mapping, by the data platform and based on the first event type, the first subset data to a first element of a data store;
mapping, by the data platform and based on the second data type, the second subset data to a second element of the data store; and
wherein the generating the graph representing activity associated with the first entity is based on the first subset data and the second subset data.
5. The method of claim 1 , wherein:
the first entity and the second entity are a same entity; and
the generating the graph representing activity associated with the first entity in the one or more cloud environments is further based on the second data stream.
6. The method of claim 1 , wherein:
the accessing the data comprises:
receiving the first data in a first data format, the first data pushed from a first data source in the one or more cloud environments; and
pulling the second data, in a second data format different from the first data format, from a second data source of the one or more cloud environments; and
the method further comprises translating the first data and the second data into a unified data format.
7. The method of claim 6 , wherein the first data is pushed by an agent deployed in the first data source.
8. The method of claim 1 , further comprising:
determining, by the data platform, a change in a quantity of the first data accessed by the data platform; and
directing, by the data platform and based on the change, the data streaming platform to scale the first data stream independent of the second data stream to maintain a threshold throughput.
9. The method of claim 1 , wherein:
the mapping the first data to the first data stream is based on a first configuration value accessed, based on the first entity and the first data type, from a key-value store; and
the mapping the second data to the second data stream is based on a second configuration value accessed, based on the second entity and the second data type, from the key-value store.
10. The method of claim 9 , wherein the key-value store comprises a hierarchical structure based on data type followed by entity.
11. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions capable of being executed to:
access data from one or more cloud environments;
identify, in the data, first data associated with a first entity and a first data type and second data associated with a second entity and a second data type;
map, based on the first entity and the first data type, the first data to a first data stream of a data streaming platform; map, based on the second entity and the second data type, the second data to a second data stream of the data streaming platform, the second data stream separate from the first data stream of the data streaming platform; and
generate, based on the first data stream, a graph representing activity associated with the first entity in the one or more cloud environments.
12. The computer program product of claim 11 , wherein a data type identifies a data source among the one or more cloud environments.
13. The computer program product of claim 11 , the computer instructions further capable of being executed to:
identify, in the first data stream, first subset data having a first event type and second subset data having a second event type;
map, based on the first event type, the first subset data to a first element of a data store;
map, based on the second data type, the second subset data to a second element of the data store; and
wherein the generating the graph representing activity associated with the first entity is based on the first subset data and the second subset data.
14. The computer program product of claim 11 , wherein:
the first entity and the second entity are a same entity; and
the generating the graph representing activity associated with the first entity in the one or more cloud environments is further based on the second data stream.
15. The computer program product of claim 11 , wherein:
the accessing the data comprises:
receiving the first data in a first data format, the first data pushed from a first data source in the one or more cloud environments; and
pulling the second data, in a second data format different from the first data format, from a second data source of the one or more cloud environments; and
the computer instructions further capable of being executed to translate the first data and the second data into a unified data format.
16. The computer program product of claim 11 , the computer instructions further capable of being executed to:
determine a change in a quantity of the first data accessed from the one or more cloud environments; and
direct, based on the change, the data streaming platform to scale the first data stream independent of the second data stream to maintain a threshold throughput.
17. The computer program product of claim 11 , wherein:
the mapping the first data to the first data stream is based on a first configuration value accessed, based on the first entity and the first data type, from a key-value store; and
the mapping the second data to the second data stream is based on a second configuration value accessed, based on the second entity and the second data type, from the key-value store.
18. A system comprising:
a processor configured to: access data from one or more cloud environments;
identify, in the data, first data associated with a first entity and a first data type and second data associated with a second entity and a second data type;
map, based on the first entity and the first data type, the first data to a first data stream of a data streaming platform;
map, based on the second entity and the second data type, the second data to a second data stream of the data streaming platform, the second data stream different separate from the first data stream of the data streaming platform; and
generate, based on the first data stream, a graph representing activity associated with the first entity in the one or more cloud environments; and a memory coupled to the processor and configured to provide the processor with instructions.
19. The system of claim 18 , wherein:
the accessing the data comprises:
receiving the first data in a first data format, the first data pushed from a first data source in the one or more cloud environments; and
pulling the second data, in a second data format different from the first data format, from a second data source of the one or more cloud environments; and
the processor is further configured to translate the first data and the second data into a unified data format.
20. The system of claim 18 , wherein the processor is further configured to:
determine a change in a quantity of the first data accessed from the one or more cloud environments; and
direct, based on the change, the data streaming platform to scale the first data stream independent of the second data stream to maintain a threshold throughput.