Network ingestion of multimodal multisource machine data to traffic regulated volume
A server group of a data intake and query system (DIQS) establishes connections with multiple source data network nodes. Data from the multiple sources comports with a variety of different data modes and may be received via the established network connections on a periodic or continuous basis for ongoing capture as modal entries of modal buckets of a common networked storage volume. Rates of data reception across the network connections influences a process to maintain a measured utilization of storage volume capacity at, near, or below a targeted level.
1. A method comprising:
receiving first-mode machine source data at a server group from a first source network node and transforming the first-mode machine source data to a plurality of entries of a first-mode storage bucket of a network storage volume having a target active data capacity, the first-mode storage bucket having first bucket metadata;
receiving second-mode machine source data at the server group from a second source network node and transforming the second-mode machine source data to a plurality of entries of a second-mode storage bucket of the network storage volume, the second-mode storage bucket having second bucket metadata;
detecting a fullness condition of the network storage volume based at least in part on the target active data capacity;
selecting at least one bucket to be purged from a plurality of buckets of the network storage volume based at least in part on bucket metadata of the plurality of buckets, the plurality of buckets including the first-mode storage bucket and the second-mode storage bucket; and
causing a purge of the at least one selected bucket from the target active data capacity of the network storage volume,
wherein the method is performed by one or more processors including at least one processor of the server group.
2. The method of claim 1 wherein receiving first-mode machine source data at a server group from a first source network node includes receiving first-mode machine source data via a first network connection established between a first source network node and a server group receiving an initial communication from the first source network node.
3. The method of claim 1 wherein receiving second-mode machine source data at a server group from a second source network node includes receiving second-mode machine source data via a second network connection established between a second source network node and a server group receiving an initial communication from the second source network node.
4. The method of claim 1 wherein the first source network node and the second source network node are each coupled to the server group by the same network.
5. The method of claim 1 wherein the first-mode machine source data is event mode data.
6. The method of claim 1 wherein the first-mode machine source data is metric mode data.
7. The method of claim 1 wherein the first-mode machine source data is event mode data and the second-mode machine source data is metric data.
8. The method of claim 1 wherein the target active data capacity is reflected in control storage of a multi-modal data input and query system.
9. The method of claim 1 wherein the target active data capacity is reflected in control storage of a multi-modal data input and query system in an encoded form.
10. The method of claim 1 wherein the target active data capacity indicates a physical storage capacity of a storage device.
11. The method of claim 1 wherein the target active data capacity indicates a measure of data associated with a particular logical state.
12. The method of claim 1 wherein the target active data capacity indicates a measure of data associated with a searchable state.
13. The method of claim 1 wherein detecting a fullness condition includes consideration of a time value associated with the first-mode storage bucket.
14. The method of claim 1 wherein detecting a fullness condition includes consideration of a time value associated with the first-mode storage bucket and a time value associated with the second-mode storage bucket.
15. The method of claim 1 wherein detecting a fullness condition includes consideration of a time value associated at least one bucket of the plurality of buckets.
16. The method of claim 1 wherein detecting a fullness condition includes consideration of a time value associated each bucket of the plurality of buckets.
17. The method of claim 1 wherein detecting a fullness condition includes a relative comparison of time values associated each of two or more buckets of the plurality of buckets.
18. The method of claim 1 wherein detecting a fullness condition includes consideration of an earliest time value associated at least one bucket of the plurality of buckets.
19. The method of claim 1 wherein detecting a fullness condition includes a relative comparison of time values each associated with a respective one of two or more buckets of the plurality of buckets, and wherein each of the time values represents an earliest time associated with its respective bucket.
20. The method of claim 1 wherein causing a purge of the at least one selected bucket includes deletion of the bucket from a filesystem.
21. The method of claim 1 wherein causing a purge of the at least one selected bucket includes changing a logical state associated with each of the at least one selected bucket.
22. The method of claim 1 wherein causing a purge of the at least one selected bucket includes changing a searchable state associated with each of the at least one selected bucket.
23. A system comprising:
a memory; and
a processing device coupled with the memory to perform operations comprising:
receiving first-mode machine source data at a server group from a first source network node and transforming the first-mode machine source data to a plurality of entries of a first-mode storage bucket of a network storage volume having a target active data capacity, the first-mode storage bucket having first bucket metadata;
receiving second-mode machine source data at the server group from a second source network node and transforming the second-mode machine source data to a plurality of entries of a second-mode storage bucket of the network storage volume, the second-mode storage bucket having second bucket metadata;
detecting a fullness condition of the network storage volume based at least in part on the target active data capacity;
selecting at least one bucket to be purged from a plurality of buckets of the network storage volume based at least in part on bucket metadata of the plurality of buckets, the plurality of buckets including the first-mode storage bucket and the second-mode storage bucket; and
causing a purge of the at least one selected bucket from the target active data capacity of the network storage volume,
wherein the operations are performed by one or more processors including at least one processor of the server group.
24. The system of claim 23 wherein causing a purge of the at least one selected bucket includes at least one from among: deletion of the bucket from a filesystem, changing a logical state associated with each of the at least one selected bucket, and changing a searchable state associated with each of the at least one selected bucket.
25. The system of claim 23 wherein detecting a fullness condition includes at least one from among: consideration of a time value associated with the first-mode storage bucket, consideration of a time value associated with the first-mode storage bucket and a time value associated with the second-mode storage bucket, consideration of a time value associated at least one bucket of the plurality of buckets, consideration of a time value associated each bucket of the plurality of buckets, a relative comparison of time values associated each of two or more buckets of the plurality of buckets, consideration of an earliest time value associated at least one bucket of the plurality of buckets, and a relative comparison of time values each associated with an earliest time associated with a respective one of two or more buckets of the plurality of buckets.
26. The system of claim 23 wherein the first-mode machine source data is event mode data and the second-mode machine source data is metric data.
27. A non-transitory computer readable storage medium encoding instructions thereon that, in response to execution by one or more processing devices, cause the one or more processing devices to perform operations comprising:
receiving first-mode machine source data at a server group from a first source network node and transforming the first-mode machine source data to a plurality of entries of a first-mode storage bucket of a network storage volume having a target active data capacity, the first-mode storage bucket having first bucket metadata;
receiving second-mode machine source data at the server group from a second source network node and transforming the second-mode machine source data to a plurality of entries of a second-mode storage bucket of the network storage volume, the second-mode storage bucket having second bucket metadata;
detecting a fullness condition of the network storage volume based at least in part on the target active data capacity;
selecting at least one bucket to be purged from a plurality of buckets of the network storage volume based at least in part on bucket metadata of the plurality of buckets, the plurality of buckets including the first-mode storage bucket and the second-mode storage bucket; and
causing a purge of the at least one selected bucket from the target active data capacity of the network storage volume,
wherein the operations are performed by one or more processors including at least one processor of the server group.
28. The non-transitory computer readable storage medium of claim 27 wherein causing a purge of the at least one selected bucket includes at least one from among: deletion of the bucket from a filesystem, changing a logical state associated with each of the at least one selected bucket, and changing a searchable state associated with each of the at least one selected bucket.
29. The non-transitory computer readable storage medium of claim 27 wherein detecting a fullness condition includes at least one from among: consideration of a time value associated with the first-mode storage bucket, consideration of a time value associated with the first-mode storage bucket and a time value associated with the second-mode storage bucket, consideration of a time value associated at least one bucket of the plurality of buckets, consideration of a time value associated each bucket of the plurality of buckets, a relative comparison of time values associated each of two or more buckets of the plurality of buckets, consideration of an earliest time value associated at least one bucket of the plurality of buckets, and a relative comparison of time values each associated with an earliest time associated with a respective one of two or more buckets of the plurality of buckets.
30. The non-transitory computer readable storage medium of claim 27 wherein the first-mode machine source data is event mode data and the second-mode machine source data is metric data.