Low overhead observability in cloud native environments

1 . A computer-implemented method, in a data processing system, for collecting telemetry data from an observability tool of a container executing on a host computing system, the computer-implemented method comprising:

configuring a shared memory to implement a telemetry data buffer, wherein

the telemetry data buffer is shared by observability tool instances of a plurality of containers on one or more host computing systems,

the plurality of containers comprises the container, and

the one or more host computing systems comprise the host computing system;

collecting, by at least one observability tool instance of the observability tool instances, the telemetry data from a corresponding container in the plurality of containers;

attempting to obtain a lock on the telemetry data buffer;

in a case where the lock is obtained on the telemetry data buffer:

writing the collected telemetry data to a next entry in the telemetry data buffer to which the collected telemetry data is writable; and

accessing, by a backend computing system, records in the telemetry data buffer to apply analytics to the collected telemetry data; and

in a case where the lock is not obtained, sending the collected telemetry data to the backend computing system via a socket based communication.

2 . The computer-implemented method of claim 1 , wherein the accessing of the records in the telemetry data buffer comprises:

retrieving, by an Agent executing on a corresponding host computing system of the one or more host computing systems, the records from the telemetry data buffer; and

sending the retrieved records to the backend computing system.

3 . The computer-implemented method of claim 1 , wherein the accessing of the records in the telemetry data buffer comprises reading the records from the telemetry data buffer using receive work queues of a remote data memory access environment.

4 . The computer-implemented method of claim 1 , wherein the writing of the collected telemetry data to the next entry in the telemetry data buffer comprises:

configuring the observability tool instances to write the collected telemetry data to the telemetry data buffer directly; and

configuring an Agent executing on the same host computing system, of the one or more host computing systems, as the observability tool instances to read the records from the telemetry data buffer.

5 . The computer-implemented method of claim 1 , wherein

the writing of the collected telemetry data to the next entry in the telemetry data buffer comprises configuring remote direct memory access (RDMA) software libraries associated with the observability tool instances to write the collected telemetry data to the telemetry data buffer using RDMA write operations, and

the telemetry data buffer is present on the backend computing system.

6 . The computer-implemented method of claim 1 , further comprising:

setting a timer based on an amount of the collected telemetry data present in the telemetry data buffer to adjust a frequency of reading the collected telemetry data from the telemetry data buffer, wherein the accessing of the records in the telemetry data buffer comprises performing a read operation each time the timer expires.

7 . The computer-implemented method of claim 6 , wherein

the timer is set to a relatively longer period of time based on the amount of the collected telemetry data reaching or falling below a low threshold, and

the timer is set to a relatively shorter period of time based on the amount of the collected telemetry data reaching or exceeding a high threshold.

8 . The computer-implemented method of claim 6 , further comprising modifying a number of work requests available in a receive request pool based on the setting of the timer.

9 . The computer-implemented method of claim 1 , wherein the at least one observability tool instance is an OpenTelemetry Software Development Kit instance.

10 . A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to collect telemetry data from an observability tool of a container executing on a host computing system, at least by:

configuring a shared memory to implement a telemetry data buffer, wherein

the telemetry data buffer is shared by observability tool instances of a plurality of containers on one or more host computing systems,

the plurality of containers comprises the container, and

the one or more host computing systems comprise the host computing system;

collecting, by at least one observability tool instance of the observability tool instances, the telemetry data from a corresponding container in the plurality of containers;

attempting to obtain a lock on the telemetry data buffer;

in a case where the lock is obtained on the telemetry data buffer:

writing the collected telemetry data to a next entry in the telemetry data buffer to which the collected telemetry data is writable; and

accessing, by a backend computing system, records in the telemetry data buffer to apply analytics to the collected telemetry data; and

in a case where the lock is not obtained, sending the collected telemetry data to the backend computing system via a socket based communication.

11 . The computer program product of claim 10 , wherein the accessing of the records in the telemetry data buffer comprises:

retrieving, by an Agent executing on a corresponding host computing system of the one or more host computing systems, the records from the telemetry data buffer; and

sending the retrieved records to the backend computing system.

12 . The computer program product of claim 10 , wherein the accessing of the records in the telemetry data buffer comprises reading the records from the telemetry data buffer using receive work queues of a remote data memory access environment.

13 . The computer program product of claim 10 , wherein the writing of the collected telemetry data to the next entry in the telemetry data buffer comprises:

configuring the observability tool instances to write the collected telemetry data to the telemetry data buffer directly; and

configuring an Agent executing on the same host computing system, of the one or more host computing systems, as the observability tool instances to read the records from the telemetry data buffer.

14 . The computer program product of claim 10 , wherein

the writing of the collected telemetry data to the next entry in the telemetry data buffer comprises configuring remote direct memory access (RDMA) software libraries associated with the observability tool instances to write the collected telemetry data to the telemetry data buffer using RDMA write operations, and

the telemetry data buffer is present on the backend computing system.

15 . The computer program product of claim 10 , wherein the computer program product further causes the computing device to:

set a timer based on an amount of the collected telemetry data present in the telemetry data buffer to adjust a frequency of reading the collected telemetry data from the telemetry data buffer, wherein the accessing of the records in the telemetry data buffer comprises performing a read operation each time the timer expires.

16 . The computer program product of claim 15 , wherein

the timer is set to a relatively longer period of time based on the amount of the collected telemetry data reaching or falling below a low threshold, and

the timer is set to a relatively shorter period of time based on the amount of the collected telemetry data reaching or exceeding a high threshold.

17 . The computer program product of claim 15 , further comprising modifying a number of work requests available in a receive request pool based on the set timer.

18 . An apparatus, comprising:

at least one processor; and

at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to collect telemetry data from an observability tool of a container executing on a host computing system, at least by:

configuring a shared memory to implement a telemetry data buffer, wherein

the telemetry data buffer is shared by observability tool instances of a plurality of containers on one or more host computing systems,

the plurality of containers comprises the container, and

the one or more host computing systems comprise the host computing system;

collecting, by at least one observability tool instance of the observability tool instances, the telemetry data from a corresponding container in the plurality of containers;

attempting to obtain a lock on the telemetry data buffer:

in a case where the lock is obtained on the telemetry data buffer:

writing the collected telemetry data to a next entry in the telemetry data buffer to which the collected telemetry data is writable; and

accessing, by a backend computing system, records in the telemetry data buffer to apply analytics to the collected telemetry data; and

in a case where the lock is not obtained, sending the collected telemetry data to the backend computing system via a socket based communication.