Hierarchichal sharding of flows from sensors to collectors
Systems, methods, and computer-readable media for hierarchichal sharding of flows from sensors to collectors. A first collector can receive a first portion of a network flow from a first capturing agent and determine that a second portion of the network flow was not received from the first capturing agent. The first collector can then send the first portion of the network flow to a second collector. A third collector can receive the second portion of the network flow from a second capturing agent and determine that the third collector did not receive the first portion of the network flow. The third collector can then send the second portion of the network flow to the second collector. The second collector can then aggregate the first portion and second portion of the network flow to yield the entire portion of the network flow.
1. A method for managing a plurality of collectors including at least first, second and third collectors to recombine a network flow having first and second portions that have become separated, the method comprising:
receiving, by the first collector, the first portion of a network flow;
first determining, by the first collector, that the second portion of the network flow was not received;
based on the first determining, sending, by the first collector, the first portion of the network flow to the second collector;
receiving, by the third collector, the second portion of the network flow;
second determining, by the third collector, that the first portion of the network flow was not received;
based on the second determining, sending, by the third collector, the second portion of the network flow to the second collector; and
combining, by the second collector, the first portion of the network flow and the second portion of the network flow.
2. The method of claim 1 , further comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collectors from the plurality of collectors.
3. The method of claim 2 , wherein the first portion of the network flow is transmitted by a first capturing agent to a first shard from the respective shards, and the second portion of the network flow is transmitted by a second capturing agent to a second shard from the respective shards, wherein the first collector is part of the first shard and the third collector is part of the second shard.
4. The method of claim 2 , wherein the second collector is part of a shard from the respective shards, and wherein the second collector is assigned a flow key and hash that corresponds to the network flow.
5. The method of claim 4 , wherein the second collector is selected to receive the first portion and the second portion of the network flow from a plurality of other collectors in the shard based on the flow key and hash corresponding to the network flow.
6. The method of claim 1 , wherein the first collector and the third collector are mapped to respective shards from a first layer of shards, each of the respective shards comprising a selected group of collectors, and wherein the second collector is mapped to a shard from a second layer of shards.
7. The method of claim 6 , wherein the first layer of shards is designated to receive network flow data directly from capturing agents deployed throughout a network, and wherein the second layer of shards is designated to receive different portions of the network flow data directly from different collectors in the first layer of shards.
8. A non-transitory computer-readable storage media storing instructions for managing a plurality of collectors including at least first, second and third collectors to recombine a network flow having first and second portions that have become separated, which when executed by a processor cause the processor to perform operations comprising:
receiving, by the first collector, the first portion of a network flow;
first determining, by the first collector, that the second portion of the network flow was not received;
based on the first determining, sending, by the first collector, the first portion of the network flow to the second collector;
receiving, by the third collector, the second portion of the network flow;
second determining, by the third collector, that the first portion of the network flow was not received;
based on the second determining, sending, by the third collector, the second portion of the network flow to the second collector; and
combining, by the second collector, the first portion of the network flow and the second portion of the network flow.
9. The media of claim 8 , the operations further comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collectors from the plurality of collectors.
10. The media of claim 9 , wherein the first portion of the network flow is transmitted by a first capturing agents to a first shard from the respective shards, and the second portion of the network flow is transmitted by a second capturing agents to a second shard from the respective shards, wherein the first collector is part of the first shard and the third collector is part of the second shard.
11. The media of claim 9 , wherein the second collector is part of a shard from the respective shards, and wherein the second collector is assigned a flow key and hash that corresponds to the network flow.
12. The media of claim 11 , wherein the second collector is selected to receive the first portion and the second portion of the network flow from a plurality of other collectors in the shard based on the flow key and hash corresponding to the network flow.
13. The media of claim 8 , wherein the first collector and the third collector are mapped to respective shards from a first layer of shards, each of the respective shards comprising a selected group of collectors, and wherein the second collector is mapped to a shard from a second layer of shards.
14. The media of claim 13 , wherein the first layer of shards is designated to receive network flow data directly from capturing agents deployed throughout a network, and wherein the second layer of shards is designated to receive different portions of the network flow data directly from different collectors in the first layer of shards.
15. A system for managing a plurality of collectors including at least first, second and third collectors to recombine a network flow having first and second portions that have become separated, the system comprising:
a processor; and
a computer-readable storage device having stored therein instructions which, when executed by the processor, cause the system to perform operations comprising:
receiving, by the first collector, the first portion of a network flow;
first determining, by the first collector, that the second portion of the network flow was not received;
based on the first determining, sending, by the first collector, the first portion of the network flow to the second collector;
receiving, by the third collector, the second portion of the network flow;
second determining, by the third collector, that the first portion of the network flow was not received;
based on the second determining, sending, by the third collector, the second portion of the network flow to the second collector; and
combining, by the second collector, the first portion of the network flow and the second portion of the network flow.
16. The system of claim 15 , the operations further comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collectors from the plurality of collectors.
17. The system of claim 16 , wherein the first portion of the network flow is transmitted by a first capturing agents to a first shard from the respective shards, and the second portion of the network flow is transmitted by a second capturing agents to a second shard from the respective shards, wherein the first collector is part of the first shard and the third collector is part of the second shard.
18. The system of claim 16 , wherein the second collector is part of a shard from the respective shards, and wherein the second collector is assigned a flow key and hash that corresponds to the network flow.
19. The system of claim 18 , wherein the second collector is selected to receive the first portion and the second portion of the network flow from a plurality of other collectors in the shard based on the flow key and hash corresponding to the network flow.
20. The system of claim 15 , wherein the first collector and the third collector are mapped to respective shards from a first layer of shards, each of the respective shards comprising a selected group of collectors, and wherein the second collector is mapped to a shard from a second layer of shards.