Hierarchichal sharding of flows from sensors to collectors
Systems, methods, and computer-readable media for hierarchichal sharding of flows from sensors to collectors. A first collector can receive a first portion of a network flow from a first capturing agent and determine that a second portion of the network flow was not received from the first capturing agent. The first collector can then send the first portion of the network flow to a second collector. A third collector can receive the second portion of the network flow from a second capturing agent and determine that the third collector did not receive the first portion of the network flow. The third collector can then send the second portion of the network flow to the second collector. The second collector can then aggregate the first portion and second portion of the network flow to yield the entire portion of the network flow.
1. A method for recombining a network flow, the method comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collectors;
receiving, by a first collector, a first portion of the network flow;
receiving, at a second collector, a second portion of the network flow from a third collector that received the second portion of the network flow but not the first portion of the network flow;
determining, by the first collector, that the second portion of the network flow was not received at the first collector;
in response to determining the second portion was not received, sending, by the first collector, the first portion of the network flow to the second collector; and
combining, by the second collector, the first portion of the network flow and the second portion of the network flow;
wherein the second collector is part of a second shard from the respective shards, and wherein the second collector is assigned a flow key and hash that corresponds to the network flow;
wherein the second collector is selected to receive the first portion and the second portion of the network flow from a plurality of other collectors based on the flow key and hash corresponding to the network flow.
2. The method of claim 1 , wherein the first portion of the network flow is transmitted by a first capturing agent to a first shard from the respective shards, wherein the first collector is part of the first shard.
3. The method of claim 1 , wherein the first collector is mapped to a respective shard from a first layer of shards, each of the respective shards comprising a selected group of collectors, and wherein the second collector is mapped to a shard from a second layer of shards.
4. A non-transitory computer-readable storage medium storing instructions to recombine a network flow, which when executed by at least one processor cause the at least one processor to perform operations comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collectors;
receiving, by a first collector, a first portion of the network flow;
receiving, at a second collector, a second portion of the network flow from a third collector that received the second portion of the network flow but not the first portion of the network flow;
determining, by the first collector, that the second portion of the network flow was not received at the first collector;
in response to determining the second portion was not received, sending, by the first collector, the first portion of the network flow to the second collector; and
combining, by the second collector, the first portion of the network flow and the second portion of the network flow;
wherein the second collector is part of a second shard from the respective shards, and wherein the second collector is assigned a flow key and hash that corresponds to the network flow;
wherein the second collector is selected to receive the first portion and the second portion of the network flow from a plurality of other collectors based on the flow key and hash corresponding to the network flow.
5. The non-transitory computer-readable storage medium of claim 4 , wherein the first portion of the network flow is transmitted by a first capturing agent to a first shard from the respective shards, wherein the first collector is part of the first shard.
6. The non-transitory computer-readable storage medium of claim 4 , wherein the first collector is mapped to a respective shard from a first layer of shards, each of the respective shards comprising a selected group of collectors, and wherein the second collector is mapped to a shard from a second layer of shards.
7. A system that recombines a network flow, the system comprising:
at least one processor; and
at least one non-transitory computer-readable storage medium having stored therein instructions, which when executed by the at least one processor, causes the at least one processor to perform operations comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collectors;
receiving, by a first collector, a first portion of the network flow;
receiving, at a second collector, a second portion of the network flow from a third collector that received the second portion of the network flow but not the first portion of the network flow;
determining, by the first collector, that the second portion of the network flow was not received at the first collector;
in response to determining the second portion was not received, sending, by the first collector, the first portion of the network flow to the second collector; and
combining, by the second collector, the first portion of the network flow and the second portion of the network flow;
wherein the second collector is part of a second shard from the respective shards, and wherein the second collector is assigned a flow key and hash that corresponds to the network flow;
wherein the second collector is selected to receive the first portion and the second portion of the network flow from a plurality of other collectors based on the flow key and hash corresponding to the network flow.
8. The system of claim 7 , wherein the first portion of the network flow is transmitted by a first capturing agent to a first shard from the respective shards, wherein the first collector is part of the first shard.
9. The system of claim 7 , wherein the first collector is mapped to a respective shard from a first layer of shards, each of the respective shards comprising a selected group of collectors, and wherein the second collector is mapped to a shard from a second layer of shards.