Methods, systems, and computer readable mediums for performing metadata-driven data collection

1. A method comprising:

receiving, by a data collection engine executing on at least one processor of a management computer system, a request for system status data for a plurality of components of a distributed computing system while the distributed computing system is in operation, wherein the request includes metadata specifying a data collection sequence for collecting component-level system status data, and wherein the plurality of components of the distributed computing system includes a plurality of compute components, a plurality of network components, and a plurality of storage components;

obtaining, by the data collection engine and using the metadata, the component-level system status data by querying a plurality of protocol-based data collectors in an order, one after the other, as specified by the data collection sequence specified by the metadata, wherein the metadata specifies, for at least some of the component-level system status data, the data collection sequence so that at least some earlier-executing data collectors are assured to execute before some later-executing data collectors;

assembling, by the data collection engine, the component-level system status data into assembled status data and storing the assembled status data in memory of the management computer system;

receiving an update to the metadata and repeating the obtaining and assembling of the component-level system status data using the update to the metadata without recompiling executable code of the protocol-based data collectors;

wherein obtaining the component-level system status data comprises querying a first protocol-based data collector and then, after querying the first protocol-based data collector, querying a second protocol-based data collector according to the data collection sequence and using the response from the first protocol-based data collector, and wherein assembling the component-level system status data comprises using a first attribute received from the first protocol-based data collector for updating an assembled object with a second attribute received from the second protocol-based data collector; and

wherein querying the first protocol-based data collector comprises receiving the first attribute and a common attribute, wherein querying the second protocol-base data collector comprises receiving the second attribute and the common attribute, and wherein updating an assembled object comprises associating the first and second attributes and the common attribute in the assembled object, and wherein the common attribute is a unique identifier of a virtual machine, and wherein updating the assembled object comprises combining the first attribute and the second attribute into a report about the virtual machine.

2. The method of claim 1 , wherein obtaining the component-level system status data comprises, in a first pass, querying a first protocol-based data collector and then a second protocol-based data collector, and then, in a second pass, querying the first protocol-based data collector using a response from the second protocol-based data collector in the first pass.

3. The method of claim 1 , wherein the metadata specifies a schedule of data collection events, and wherein querying the protocol-based data collectors in the order as specified by the data collection sequence comprises querying the protocol-based data collectors in response to determining that a current time matches a scheduled data collection event.

4. A system comprising:

at least one processor and memory; and

a data collection engine implemented using the at least one processor, wherein the data collection engine is programmed for:

receiving a request for system status data for a plurality of components of a distributed computing system while the distributed computing system is in operation, wherein the request includes metadata specifying a data collection sequence for collecting component-level system status data, and wherein the plurality of components of the distributed computing system includes a plurality of compute components, a plurality of network components, and a plurality of storage components;

obtaining, using the metadata, the component-level system status data by querying a plurality of protocol-based data collectors in an order, one after the other, as specified by the data collection sequence specified by the metadata, wherein the metadata specifies, for at least some of the component-level system status data, the data collection sequence so that at least some earlier-executing data collectors are assured to execute before some later-executing data collectors; and

assembling the component-level system status data into assembled status data and storing the assembled status data in the memory;

wherein obtaining the component-level system status data comprises querying a first protocol-based data collector and then, after querying the first protocol-based data collector, querying a second protocol-based data collector according to the data collection sequence and using the response from the first protocol-based data collector, and wherein assembling the component-level system status data comprises using a first attribute received from the first protocol-based data collector for updating an assembled object with a second attribute received from the second protocol-based data collector;

wherein querying the first protocol-based data collector comprises receiving the first attribute and a common attribute, wherein querying the second protocol-base data collector comprises receiving the second attribute and the common attribute, and wherein updating an assembled object comprises associating the first and second attributes and the common attribute in the assembled object; and

wherein the data collection engine is programmed for receiving an update to the metadata and repeating the obtaining and assembling of the component-level system status data using the update to the metadata without recompiling executable code of the protocol-based data collectors, and wherein the common attribute is a unique identifier of a virtual machine, and wherein updating the assembled object comprises combining the first attribute and the second attribute into a report about the virtual machine.

5. The system of claim 4 , wherein obtaining the component-level system status data comprises, in a first pass, querying a first protocol-based data collector and then a second protocol-based data collector, and then, in a second pass, querying the first protocol-based data collector using a response from the second protocol-based data collector in the first pass.

6. The system of claim 4 , wherein the metadata specifies a schedule of data collection events, and wherein querying the protocol-based data collectors in the order as specified by the data collection sequence comprises querying the protocol-based data collectors in response to determining that a current time matches a scheduled data collection event.

7. A non-transitory computer readable medium storing executable instructions that, when executed by at least one processor of a computer system, cause the computer system to perform operations comprising:

receiving a request for system status data for a plurality of components of a distributed computing system while the distributed computing system is in operation, wherein the request includes metadata specifying a data collection sequence for collecting component-level system status data, and wherein the plurality of components of the distributed computing system includes a plurality of compute components, a plurality of network components, and a plurality of storage components;

obtaining, using the metadata, the component-level system status data by querying a plurality of protocol-based data collectors in an order, one after the other, as specified by the data collection sequence specified by the metadata, wherein the metadata specifies, for at least some of the component-level system status data, the data collection sequence so that at least some earlier-executing data collectors are assured to execute before some later-executing data collectors; and

assembling the component-level system status data into assembled status data and storing the assembled status data in memory of the computer system;

wherein obtaining the component-level system status data comprises querying a first protocol-based data collector and then, after querying the first protocol-based data collector, querying a second protocol-based data collector according to the data collection sequence and using the response from the first protocol-based data collector, and wherein assembling the component-level system status data comprises using a first attribute received from the first protocol-based data collector for updating an assembled object with a second attribute received from the second protocol-based data collector;

wherein querying the first protocol-based data collector comprises receiving the first attribute and a common attribute, wherein querying the second protocol-base data collector comprises receiving the second attribute and the common attribute, and wherein updating an assembled object comprises associating the first and second attributes and the common attribute in the assembled object; and

wherein the operations comprise receiving an update to the metadata and repeating the obtaining and assembling of the component-level system status data using the update to the metadata without recompiling executable code of the protocol-based data collectors, and wherein the common attribute is a unique identifier of a virtual machine, and wherein updating the assembled object comprises combining the first attribute and the second attribute into a report about the virtual machine.

8. The non-transitory computer readable medium of claim 7 , wherein the metadata specifies a schedule of data collection events, and wherein querying the protocol-based data collectors in the order as specified by the data collection sequence comprises querying the protocol-based data collectors in response to determining that a current time matches a scheduled data collection event.