METHODS AND SYSTEMS THAT PROVIDE EFFICIENT CONTENTION CONTROL IN DISTRIBUTED COMPUTER SYSTEMS
The current document is directed to contention control for computational resources in distributed computer systems and, in particular, to contention control for memory in distributed metrics collection systems that collect and aggregate metric data in distributed computer systems. In one implementation, parallel metric-data collectors in a first distributed computer system collect metric data and one or more aggregators aggregate collected metric data and forward the aggregated metric data to a second distributed computer system, which uses the metric data for various monitoring, analysis, and management tasks. Each parallel data collector stores received metrics in a metrics container assigned to the parallel collector and a write/read-write lock provides contention control that allows multiple metric-data collectors to concurrently access metrics containers but only a single aggregator to access the metrics containers.
1 . A distributed computer system comprising:
a management distributed computer system that receives aggregated metrics messages from one or more client distributed computer systems; and
a client distributed computer system that includes
multiple metrics collectors, each metrics collector receiving and appending metrics messages to a metrics buffer associated with the metrics collector to which only the metrics collector appends metrics messages,
one or more aggregators that
copy metrics from the metrics buffers associated with metrics collectors into a copy buffer,
aggregate metrics messages within the copy buffer, and
transmit aggregated metrics messages to the management distributed computer system, and
a write/read-write lock that controls contention for the metrics buffers associated with the multiple metrics collectors.
2 . The distributed computer system of claim 1 wherein the management distributed computer system stores the received aggregated metrics messages in a distributed database and subsequently uses the aggregated metrics messages to manage the client distributed computer system.
3 . The distributed computer system of claim 1 wherein the client distributed computer system comprises multiple execution environments, each implemented on one or more physical or virtual servers, that support execution of services and applications.
4 . The distributed computer system of claim 3 wherein each execution environment includes a service proxy that
receives service calls from applications;
forwards the received service calls to services;
receives responses to forwarded service calls from services; and
returns the received responses to the applications and transmits a metrics message to one of the metrics collectors.
5 . The distributed computer system of claim 1 wherein a metrics message includes:
a timestamp;
a source field that contains an indication of the source of a service call;
a service field that contains an indication of a service to which the service call was directed; and
one or more metric fields that each contains a value from which a metric value is derived.
6 . The distributed computer system of claim 5 wherein a set of metrics messages is aggregated to generate a corresponding aggregated metrics message by:
selecting one or more fields as aggregation fields;
for each aggregation field, selecting a set of classes or quantized values for the metric value derived from the field;
partitioning the set of metrics messages into one or more subsets, based on values of the fields in the metrics messages not selected as aggregation fields; and
representing each subset of metrics messages by an aggregated metrics message that includes a count field for each class or quantized value.
7 . The distributed computer system of claim 1 wherein the write/read-write lock provides:
a collector-lock method;
a collector-unlock method;
an aggregator-lock method; and
an aggregator-unlock method.
8 . The distributed computer system of claim 7
wherein multiple metrics collectors can acquire and concurrently hold the write/read-write lock by calling the collector-lock method; and
wherein no aggregator can acquire the write/read-write lock when one or more metrics collectors currently hold the write/read-write lock.
9 . The distributed computer system of claim 8
wherein, when an aggregator currently holds the write/read-write lock, execution of a metrics collector that calls the collector-lock method is suspended until the write/read-write lock becomes available for acquisition to metrics collectors.
10 . The distributed computer system of claim 8
wherein one aggregator can acquire and concurrently hold the write/read-write lock by calling the aggregator-lock method; and
wherein no additional aggregator can acquire or hold the write/read-write lock when an aggregator currently holds the write/read-write lock.
11 . The distributed computer system of claim 10
wherein, when an aggregator or one or more metrics collectors currently holds the write/read-write lock, execution of an aggregator that calls the aggregator-lock method is suspended until the write/read-write lock becomes available for acquisition to aggregators.
12 . The distributed computer system of claim 10
wherein, when one or metrics collectors concurrently hold the write/read-write lock and an aggregator calls the aggregator-lock method an execution of the aggregator is suspended, additional metrics collectors cannot acquire and hold the write/read-write lock until the suspended aggregator acquires and holds the write/read-write lock.
13 . The distributed computer system of claim 1
wherein each metrics collector, following initialization, waits to receive metrics messages; and
wherein, when a metrics collector has received at least one metrics message, the metrics collector
calls a collector-lock method of the write/read-write lock to acquire and hold the write/read-write lock,
appends one or more received metrics messages to the metrics buffer associated with the metrics collector, and
calls a collector-unlock method of the write/read-write lock to release the write/read-write lock.
14 . The distributed computer system of claim 1
wherein an aggregator, following initialization, waits for a next aggregation time interval to begin; and
wherein, when a next aggregation time interval has begun, the aggregator
calls an aggregator-lock method of the write/read-write lock to acquire and hold the write/read-write lock,
copies metrics messages from the metrics buffers associated with the metrics collectors into a copy buffer,
reinitializes the metrics buffers associated with the metrics collectors,
calls an aggregator-unlock method of the write/read-write lock to release the write/read-write lock,
aggregates metrics messages in the copy buffer to generate one or more aggregated metrics messages, and
transmits the one or more aggregated metrics messages to the first distributed computer system.
15 . A method that collects metrics messages from a client distributed computer system, aggregates the collected metrics messages to generate one or more aggregated metrics messages, and transmits the one or more aggregated metrics messages to a management distributed computer system, the method comprising:
providing, in the client distributed computer system, multiple metrics collectors and one or more aggregators;
initializing the multiple metrics collectors and one or more aggregators;
receiving, by each metric collector, metrics messages and appending the received metrics messages to a metrics buffer associated with the metrics collector and to which only the metrics collector appends metrics messages; and
periodically
copying, by an aggregator, metrics from the metrics buffers associated with metrics collectors into a copy buffer,
aggregating, by the aggregator, metrics messages within the copy buffer to generate one or more aggregated metrics messages, and
transmitting, by the aggregator, the one or more aggregated metrics messages to the management distributed computer system.
16 . The method of claim 15 further comprising:
storing, by the management distributed computer system, received aggregated metrics messages in a distributed database; and
using the stored aggregated metrics messages to manage the one or more client distributed computer systems.
17 . The method of claim 15 further comprising:
providing a write/read-write lock to control contention for metrics buffer by the multiple metrics collectors and one or more aggregators; and
wherein the write/read-write lock provides
a collector-lock method,
a collector-unlock method,
an aggregator-lock method, and
an aggregator-unlock method.
18 . The method of claim 17
wherein multiple metrics collectors can acquire and concurrently hold the write/read-write lock by calling the collector-lock method;
wherein no aggregator can acquire the write/read-write lock when one or more metrics collectors currently holds the write/read-write lock;
wherein, when an aggregator currently holds the write/read-write lock, execution of a metrics collector that calls the collector-lock method is suspended until the write/read-write lock becomes available for acquisition to metrics collectors;
wherein one aggregator can acquire and concurrently hold the write/read-write lock by calling the aggregator-lock method;
wherein no additional aggregator can acquire or hold the write/read-write lock when an aggregator currently holds the write/read-write lock;
wherein, when an aggregator or one or more metrics collectors currently holds the write/read-write lock, execution of an aggregator that calls the aggregator-lock method is suspended until the write/read-write lock becomes available for acquisition to aggregators; and
wherein, when one or metrics collectors concurrently hold the write/read-write lock and an aggregator calls the aggregator-lock method an execution of the aggregator is suspended, additional metrics collectors cannot acquire and hold the write/read-write lock until the suspended aggregator acquires and holds the write/read-write lock.
19 . The method of claim 17
wherein each metrics collector, following initialization, waits to receive metrics messages;
wherein, when a metrics collector has received at least one metrics message, the metrics collector
calls a collector-lock method of the write/read-write lock to acquire and hold the write/read-write lock,
appends one or more received metrics messages to the metrics buffer associated with the metrics collector, and
calls a collector-unlock method of the write/read-write lock to release the write read-write lock;
wherein an aggregator, following initialization, waits for a next aggregation time interval to begin; and
wherein, when a next aggregation time interval has begun, the aggregator
calls an aggregator-lock method of the write/read-write lock to acquire and hold the write/read-write lock,
copies metrics messages from the metrics buffers associated with the metrics collectors into a copy buffer,
reinitializes the metrics buffers associated with the metrics collectors,
calls an aggregator-unlock method of the write/read-write lock to release the write/read-write lock,
aggregates metrics messages in the copy buffer to generate one or more aggregated metrics messages, and
transmits the one or more aggregated metrics messages to the first distributed computer system.
20 . A physical data-storage device encoded with computer instructions that, when executed by computational resources of a distributed computer system, controls the distributed computer system to:
initialize multiple metrics collectors and one or more aggregators;
initialize a write/read-write lock to control contention for metrics buffers;
receive, by each metrics collector, metrics messages;
append, by each metrics collector, the received metrics messages to a metrics buffer associated with the metrics collector and to which only the metrics collector appends metrics messages;
periodically copying, by an aggregator, metrics from the metrics into a copy buffer,
aggregating, by the aggregator, metrics messages within the copy buffer to generate one or more aggregated metrics messages, and
transmitting, by the aggregator, the one or more aggregated metrics messages to the management distributed computer system.