Reducing amounts of data ingested into a data warehouse
Reducing amounts of data ingested into a data warehouse, including: determining a duration associated with a subject described by event data provided to a data ingestion pipeline; determining, based on the duration, an interval for providing next event data; and providing, based on the interval, the next event data.
1. A method of reducing amounts of data ingested into a data warehouse, the method comprising:
determining, by one of an agent of a compute asset being executed by a computing system and an event data collector coupled to the agent, a duration associated with a subject described by event data provided to a data ingestion pipeline;
determining, by the one of the agent of the compute asset being executed by the computing system and the event data collector coupled to the agent, based on the duration, an interval for providing next event data; and
providing, by the one of the agent of the compute asset being executed by the computing system and the event data collector coupled to the agent, based on the interval, the next event data.
2. The method of claim 1 , wherein the subject comprises a network connection and the duration comprises a duration of the network connection.
3. The method of claim 1 , wherein the subject comprises a process and the duration comprises a duration that the process has been executing by the computing system.
4. The method of claim 1 , wherein the interval is determined as a function of the duration.
5. The method of claim 1 , wherein the interval is determined based on which of one or more thresholds are exceeded by the duration.
6. The method of claim 1 , wherein determining, based on the duration, the interval is performed in response to the subject corresponding to a particular subject type.
7. The method of claim 1 , further comprising generating consolidated event data describing a plurality of network connection events.
8. The method of claim 1 , wherein two or more portions of event data comprises two portions of event data each generated by a different network connection endpoint in a same host computing system.
9. The method of claim 1 , wherein the two or more portions of event data describe two or more network connections between a same pair of network connection endpoints during the interval.
10. A method of reducing amounts of data ingested into a data warehouse, the method comprising:
identifying, by one of an agent of a compute asset being executed by a computing system and an event data collector coupled to the agent, two or more portions of event data directed to a data ingestion pipeline and describing a same pair of network connection endpoints, wherein the two or more portions of event data describe two or more network connections between the same pair of network connection endpoints during an interval;
consolidating, by the one of the agent of the compute asset being executed by the computing system and the event data collector coupled to the agent, the two or more portions of event data into fewer portions of event data by generating, as the fewer portions of event data, a portion of event data describing the two or more network connections during the interval; and
providing, by the one of the agent of the compute asset being executed by the computing system and the event data collector coupled to the agent, the fewer portions of event data to the data ingestion pipeline.
11. The method of claim 10 , wherein the two or more portions of event data comprises two portions of event data each generated by a different network connection endpoint in a same host.
12. The method of claim 11 , wherein consolidating the two or more portions of event data into fewer portions of event data comprises selecting, as the fewer portions of event data, a single portion of event data from the two portions of event data.
13. The method of claim 10 , wherein the portion of event data describing the two or more network connections during the interval indicates a number of network connections that occurred during the interval.
14. The method of claim 10 , wherein the portion of event data describing the two or more network connections during the interval indicates two or more ports used for the two or more network connections.
15. The method of claim 10 , wherein the two or more portions of event data describe a same network connection, and the method further comprises:
determining a duration of the same network connection;
determining, based on the duration, an interval; and
wherein providing the fewer portions of event data to the data ingestion pipeline comprises providing, based on the interval, the fewer portions of event data to the data ingestion pipeline.
16. A computer program product for reducing amounts of data ingested into a data warehouse, the computer program product disposed on a computer readable medium, the computer program product including computer program instructions configurable to carry out the steps of:
determining, by one of an agent of a compute asset being executed by a computing system and an event data collector coupled to the agent a duration associated with a subject described by event data provided to a data ingestion pipeline;
determining, by the one of the agent of the compute asset being executed by the computing system and the event data collector coupled to the agent, based on the duration, an interval for providing next event data; and
providing, by the one of the agent of the compute asset being executed by the computing system and the event data collector coupled to the agent, based on the interval, the next event data.
17. The computer program product of claim 16 , wherein the subject comprises a network connection and the duration comprises a duration of the network connection.
18. The computer program product of claim 16 , wherein the subject comprises a process and the duration comprises a duration that the process has been executing by the computing system.