Dynamic assignment of collector modules to multi-tenanted isolated execution environments
A host computing device can include a plurality of isolated execution environments and a communication buffer. Each of the isolated execution environments can include an application interface, a plurality of collector modules, a coordinator module, and a scheduler module. The application interface can determine configurations for a plurality of connectors configured to collect data from respective data sources. The plurality of collector modules can include collector modules that correspond to one or more connectors and collect data from data sources of one or more tenants. The coordinator module can monitor execution of the plurality of collector modules and the scheduler module can assign collector modules of a connector between the plurality of isolated execution environments. The communication buffer can include configuration data, connector-isolated execution environment assignments, and a status of collector modules.
1 . A host computing device, comprising:
a plurality of isolated execution environments configured as data collection nodes, wherein a first isolated execution environment of the plurality of isolated execution environments comprises:
a first application interface configured to determine configurations for a plurality of connectors, each connector of the plurality of connectors comprising a respective cluster of collector modules configured to collect data from a same data source,
a first plurality of collector modules, wherein a first collector module of the first plurality of collector modules is configured to collect data from a first data source of a first tenant and a second collector module of the first plurality of collector modules is configured to collect data from a first data source of a second tenant, wherein the first collector module corresponds to a first connector of the plurality of connectors and the second collector module corresponds to a second connector of the plurality of connectors,
a first coordinator module configured to monitor execution of the first plurality of collector modules within the first isolated execution environment, and
a first scheduler module configured to assign a first cluster of collector modules of the first connector between the plurality of isolated execution environments; and
a communication buffer communicatively coupled with the plurality of isolated execution environments, the communication buffer comprising:
configuration data indicating the configurations for the plurality of connectors, wherein the configuration data is editable by a plurality of application interfaces of the plurality of isolated execution environments,
assignment data indicating assignment of the first cluster of collector modules of the first connector between the plurality of isolated execution environments, wherein the assignment data is editable by the first scheduler module, and
status data indicating a status of collector modules of the plurality of connectors, wherein the status data is editable by a plurality of coordinator modules of the plurality of isolated execution environments.
2 . A method, comprising:
determining, by a first application interface of a first isolated execution environment of a plurality of isolated execution environments, configurations for a plurality of connectors, each connector of the plurality of connectors comprising a respective cluster of collector modules configured to collect data from a same data source, wherein the plurality of isolated execution environments are configured as data collection nodes;
assigning, by a first scheduler module of the first isolated execution environment, a first cluster of collector modules of a first connector of the plurality of connectors between the plurality of isolated execution environments, wherein a communication buffer is communicatively coupled with the plurality of isolated execution environments and comprises configuration data indicating the configurations for the plurality of connectors, assignment data indicating assignment of the first cluster of collector modules of the first connector between the plurality of isolated execution environments, and status data indicating a status of collector modules of the plurality of connectors, wherein the configuration data is editable by a plurality of application interfaces of the plurality of isolated execution environments, the assignment data is editable by the first scheduler module, and the status data is editable by a plurality of coordinator modules of the plurality of isolated execution environments;
collecting, by a first collector module of a first plurality of collector modules of the first isolated execution environment, data from a first data source of a first tenant;
collecting, by a second collector module of the first plurality of collector modules, data from a first data source of a second tenant; and
monitoring, by a first coordinator module of the first isolated execution environment, execution of the first plurality of collector modules within the first isolated execution environment.
3 . The method of claim 2 , wherein the plurality of isolated execution environments share compute resources of a host computing device.
4 . The method of claim 2 , wherein a third collector module of the first plurality of collector modules corresponds to the first connector and is configured to collect data from the first data source of the first tenant.
5 . The method of claim 2 , wherein a second isolated execution environment of the plurality of isolated execution environments comprises:
a second plurality of collector modules, wherein at least one collector module of the second plurality of collector modules corresponds to the first connector.
6 . The method of claim 2 , wherein a second isolated execution environment of the plurality of isolated execution environments comprises:
a second plurality of collector modules, wherein at least one collector module of the second plurality of collector modules corresponds to the first connector, and
a second coordinator module of the plurality of coordinator modules, the second coordinator module configured to monitor execution of the second plurality of collector modules within the second isolated execution environment.
7 . The method of claim 2 , wherein a second isolated execution environment of the plurality of isolated execution environments comprises:
a second plurality of collector modules, wherein at least one collector module of the second plurality of collector modules corresponds to the first connector,
a second coordinator module of the plurality of coordinator modules, the second coordinator module configured to monitor execution of the second plurality of collector modules within the second isolated execution environment, and
a second scheduler module, wherein the second scheduler module is disabled.
8 . The method of claim 2 , wherein the first scheduler module is configured to read the status data from the communication buffer.
9 . The method of claim 2 , wherein the first scheduler module is configured to read the status data from the communication buffer and modify the assignment data based on the status data.
10 . The method of claim 2 , wherein the first scheduler module is configured to read the status data from the communication buffer and modify the assignment data based on the status data, wherein to modify the assignment data, the first scheduler module is configured to modify an assignment of a third collector module from the first isolated execution environment to an assignment of the third collector module to a second isolated execution environment of the plurality of isolated execution environments.
11 . The method of claim 2 , wherein the first scheduler module is configured to read the status data from the communication buffer and modify the assignment data based on the status data to obtain modified assignment data,
wherein to modify the assignment data, the first scheduler module is configured to modify an assignment of a third collector module from the first isolated execution environment to a second isolated execution environment of the plurality of isolated execution environments to obtain a modified assignment of the third collector module, and
wherein the first coordinator module is configured to read the modified assignment data and terminate the third collector module on the first isolated execution environment based on the modified assignment of the third collector module.
12 . The method of claim 2 , wherein the first scheduler module is configured to read the status data from the communication buffer and modify the assignment data based on the status data to obtain modified assignment data,
wherein to modify the assignment data, the first scheduler module is configured to modify an assignment of a third collector module from the first isolated execution environment to a second isolated execution environment of the plurality of isolated execution environments to obtain a modified assignment of the third collector module,
wherein the first coordinator module is configured to read the modified assignment data and terminate the third collector module on the first isolated execution environment based on the modified assignment of the third collector module, and
wherein a second coordinator module of the plurality of coordinator modules is configured to read the modified assignment data and generate the third collector module on a second isolated execution environment based on the modified assignment of the third collector module, wherein the second coordinator module is configured to monitor execution of a second plurality of collector modules within the second isolated execution environment.
13 . The method of claim 2 , wherein the first scheduler module is configured to read the status data from the communication buffer and modify the assignment data based on the status data to obtain modified assignment data,
wherein to modify the assignment data, the first scheduler module is configured to modify an assignment of a third collector module from the first isolated execution environment to a second isolated execution environment of the plurality of isolated execution environments to obtain a modified assignment of the third collector module,
wherein the first coordinator module is configured to read the modified assignment data and terminate the third collector module on the first isolated execution environment based on the modified assignment of the third collector module, and
wherein a second coordinator module of the plurality of coordinator modules is configured to read the modified assignment data and the status data, determine the status data indicates reassignment of the third collector module is available based on termination of the third collector module on the first isolated execution environment, and generate the third collector module on a second isolated execution environment based on the modified assignment of the third collector module, wherein the second coordinator module is configured to monitor execution of a second plurality of collector modules within the second isolated execution environment.
14 . The method of claim 2 , wherein the first application interface is further configured to update the configuration data.
15 . The method of claim 2 , wherein the first scheduler module is further configured to obtain the configuration data and determine a number of collector modules based on the configuration data.
16 . The method of claim 2 , wherein the first scheduler module is further configured to obtain the configuration data and determine a number of collector modules based on the configuration data, wherein the first scheduler module is configured to define the first cluster of collector modules based on the number of collector modules.
17 . The method of claim 2 , wherein the first scheduler module is further configured to obtain the configuration data and update the assignment data based on the configuration data.
18 . The method of claim 2 , wherein the first scheduler module is further configured to obtain the configuration data and update the assignment data based on the configuration data to obtain updated assignment data, and
wherein the first coordinator module is further configured to read the updated assignment data, generate a third collector module based on updated assignment data, and update the status data.
19 . The method of claim 2 , wherein the first scheduler module is further configured to obtain the configuration data and update the assignment data based on the configuration data to obtain updated assignment data,
wherein the first coordinator module is further configured to read the updated assignment data, generate a third collector module on the first isolated execution environment based on the updated assignment data, and update the status data, and
wherein a second coordinator module of the plurality of coordinator modules is configured to read the updated assignment data, generate a fourth collector module on a second isolated execution environment based on the updated assignment data, and update the status data, wherein the second coordinator module is configured to monitor execution of a second plurality of collector modules within the second isolated execution environment.
20 . Non-transitory computer-readable media including computer-executable instructions that, when executed by a host computing device, cause the host computing device to:
determine, by a first application interface of a first isolated execution environment of a plurality of isolated execution environments, configurations for a plurality of connectors, each connector of the plurality of connectors comprising a respective cluster of collector modules configured to collect data from a same data source, wherein the plurality of isolated execution environments are configured as data collection nodes;
assign, by a first scheduler module of the first isolated execution environment, a first cluster of collector modules of a first connector of the plurality of connectors between the plurality of isolated execution environments, wherein a communication buffer is communicatively coupled with the plurality of isolated execution environments and comprises configuration data indicating the configurations for the plurality of connectors, assignment data indicating assignment of the first cluster of collector modules of the first connector between the plurality of isolated execution environments, and status data indicating a status of collector modules of the plurality of connectors, wherein the configuration data is editable by a plurality of application interfaces of the plurality of isolated execution environments, the assignment data is editable by the first scheduler module, and the status data is editable by a plurality of coordinator modules of the plurality of isolated execution environments;
collect, by a first collector module of a first plurality of collector modules of the first isolated execution environment, data from a first data source of a first tenant;
collect, by a second collector module of the first plurality of collector modules, data from a first data source of a second tenant; and
monitor, by a first coordinator module of the first isolated execution environment, execution of the first plurality of collector modules within the first isolated execution environment.