Time-series data monitoring with sharded server
Described are aspects of a sharded monitoring system that collects metrics data from a plurality of targets within a monitored system. In various embodiments, the monitoring system includes multiple monitoring-server instances each collecting metrics data from a respective subset of the targets, as well as a federation server that collects aggregated metrics data from the other monitoring-server instances. Mappings between monitoring targets and monitoring-server instances collecting their metrics data may be stored in a temporal routing map, e.g., on the federation server, which may build the map, in accordance with some embodiments, based on time-series routing metrics received from the other monitoring-serve instances. The routing map may be used to direct queries for metrics data from a client to the appropriate monitoring-server instance. Further aspects and embodiments are disclosed.
1. A server comprising:
one or more processors; and
memory storing:
a temporal routing map comprising routing data specifying time series of mappings between a plurality of targets and a plurality of monitoring-server instances that collect metrics data from the targets, each target corresponding to a component or a component-shard of a monitored system; and
instructions for execution by the one or more processors, the instructions, when executed, causing the one or more processors to perform operations comprising:
receiving, from a client, a routing query specifying a time and a target identifier;
determining a monitoring-server-instance identifier associated with a specified target identifier and with a timestamp corresponding to a specified time using the temporal routing map; and
sending a response to the client, the response including the monitoring-server-instance identifier associated with the specified target identifier and with the timestamp corresponding to the specified time.
2. The server of claim 1 , wherein the operations further comprise receiving messages from the monitoring-server instances as time-series routing metrics specifying one or more of the targets from which a respective monitoring-server instance is collecting the metrics data.
3. The server of claim 1 , wherein:
a mapping of a target changes at a switch time from a first monitoring-server instance to a second-monitoring server instance,
messages received from the monitoring-server instances include one or more first messages having timestamps corresponding to times preceding the switch time, a target identifier associated with the target, and a first monitoring-server-instance identifier associated with the first monitoring-server instance, and
the messages include one or more second messages having timestamps corresponding to times following the switch time, the target identifier associated with the target, and a second monitoring-server-instance identifier associated with the second monitoring-server instance.
4. The server of claim 3 , wherein the monitoring-server instances change in number at the switch time.
5. The server of claim 3 , wherein the routing query specifies the target identifier associated with the target and a time range spanning times preceding and following the switch time, and wherein the response includes the first monitoring-server-instance identifier in association with a first partial time range ending at the switch time and the second monitoring-server-instance identifier in association with a second partial time range beginning at the switch time.
6. The server of claim 5 , wherein the operations further comprise:
splitting the routing query into a first sub-query associated with the first partial time range and a second sub-query associated with the second partial time range;
sending the first sub-query to the first monitoring-server instance and receiving, in response, first metrics data from the first monitoring-server instance;
sending the second sub-query to the second monitoring-server instance and receiving, in response, second metrics data from the second monitoring-server instance; and
merging the first and second metrics data.
7. The server of claim 3 , wherein, prior to the switch time:
the first monitoring-server instance collects the metrics data from the target and from one or more additional targets, and aggregates collected metrics data into first lower-level aggregated metrics data; and
the operations further comprise receiving the first lower-level aggregated metrics data from the first monitoring-server instance and receiving additional lower-level aggregated metrics data from one or more additional monitoring-server instances, and aggregating the first and additional lower-level aggregated metrics data into higher-level aggregated metrics data.
8. The server of claim 1 , wherein the routing query is received responsive to user input received via a user interface of a visualization tool to request information about the monitored system.
9. The server of claim 1 , wherein the metrics data includes at least one of time-series operating-system metrics or time-series virtual runtime metrics.
10. The server of claim 1 , wherein the operations further comprise removing the mappings that are older than a retention period from the temporal routing map.
11. The server of claim 1 , wherein each of the monitoring-server instances is assigned to collect the metrics data from a subset of the targets.
12. A method comprising:
storing a temporal routing map comprising routing data specifying time series of mappings between a plurality of targets and a plurality of monitoring-server instances that collect metrics data from the targets, each target corresponding to a component or a component-shard of a monitored system;
receiving a routing query from a client, the routing query specifying a time and a target identifier;
determining a monitoring-server-instance identifier associated with a specified target identifier and with a timestamp corresponding to a specified time using the temporal routing map; and
sending a response to the client, the response including the monitoring-server-instance identifier associated with the specified target identifier and with the timestamp corresponding to the specified time.
13. The method of claim 12 , further comprising receiving messages from the monitoring-server instances as time-series routing metrics specifying one or more of the targets from which a respective monitoring-server instance is collecting the metrics data.
14. The method of claim 13 , wherein:
a mapping of a target changes at a switch time from a first monitoring-server instance to a second-monitoring server instance,
messages received from the monitoring-server instances include one or more first messages having timestamps corresponding to times preceding the switch time, a target identifier associated with the target, and a first monitoring-server-instance identifier associated with the first monitoring-server instance, and
the messages include one or more second messages having timestamps corresponding to times following the switch time, the target identifier associated with the target, and a second monitoring-server-instance identifier associated with the second monitoring-server instance.
15. The method of claim 14 , wherein the monitoring-server instances change in number at the switch time.
16. The method of claim 14 , wherein the routing query specifies the target identifier associated with the target and a time range spanning times preceding and following the switch time, and wherein the response includes the first monitoring-server-instance identifier in association with a first partial time range ending at the switch time and the second monitoring-server-instance identifier in association with a second partial time range beginning at the switch time.
17. The method of claim 16 , further comprising:
splitting the routing query into a first sub-query associated with the first partial time range and a second sub-query associated with the second partial time range;
sending the first sub-query to the first monitoring-server instance and receiving, in response, first metrics data from the first monitoring-server instance;
sending the second sub-query to the second monitoring-server instance and receiving, in response, second metrics data from the second monitoring-server instance; and
merging the first and second metrics data.
18. The method of claim 14 , wherein, prior to the switch time, the first monitoring-server instance collects the metrics data from the target and from one or more additional targets and aggregates collected metrics data into first lower-level aggregated metrics data, the method further comprising receiving the first lower-level aggregated metrics data from the first monitoring-server instance and receiving additional lower-level aggregated metrics data from one or more additional monitoring-server instances, and aggregating the first and additional lower-level aggregated metrics data into higher-level aggregated metrics data.
19. The method of claim 18 , wherein the targets comprise database shards, and wherein the first monitoring-server instance and the additional monitoring server instances collect metrics data from the database shards that are associated with a common keyspace.
20. One or more machine-readable media storing instructions for execution by one or more processors, the instructions, when executed, causing the one or more processors to perform operations comprising:
creating a temporal routing map comprising routing data specifying time series of mappings between a plurality of targets and a plurality of monitoring-server instances that collect metrics data from the targets, each target corresponding to a component or a component-shard of a monitored system;
receiving a routing query from a client, the routing query specifying a time and a target identifier;
determining a monitoring-server-instance identifier associated with a specified target identifier and with a timestamp corresponding to a specified time using the temporal routing map; and
sending a response to the client, the response including the monitoring-server-instance identifier associated with the specified target identifier and with the timestamp corresponding to the specified time.