Method and system for optimizing access to data nodes of a data cluster using a data access gateway and metadata mapping based bidding
A method that is performed to access data nodes of a data cluster. The method includes obtaining, by a data access gateway (DAG), a request from a host; and in response to the request, obtaining bidding counters from the data nodes; obtaining metadata mappings from the data nodes; identifying, based on the bidding counters and metadata mappings, a data node of the data nodes associated with a highest bidding counter of the bidding counters and an appropriate metadata mapping of the metadata mappings; and sending the request to the data node.
1. A method for accessing data nodes of a data cluster, the method comprising:
obtaining, by a data access gateway (DAG), a request from a host; and
in response to the request:
obtaining bidding counters from data nodes, wherein the bidding counters comprise numerical representations of computational, network, and storage resources of the data nodes available to service the request;
obtaining metadata mappings from the data nodes, wherein:
a metadata mapping of the metadata mappings comprises a data node identifier associated with a data node of the data nodes corresponding to the metadata mapping and a plurality of chunk metadata; and
a chunk metadata of the plurality of chunk metadata may specify:
a chunk identifier associated with a corresponding chunk,
whether the chunk is a data chunk or a parity chunk,
whether the chunk is stored in an accelerator pool or a non-accelerator pool,
a storage location of the chunk, and
a data slice identifier associated with the chunk;
identifying, based on the bidding counters and metadata mappings, a data node of the data nodes associated with a highest bidding counter of the bidding counters and an appropriate metadata mapping of the metadata mappings, wherein the data node is associated with a highest amount of available computational, network, and storage resources, as specified by the highest bidding counter, compared to other data nodes of the data nodes; and
sending the request to the data node.
2. The method of claim 1 , wherein a bidding counter of the bidding counters is associated with a data node of the data nodes.
3. The method of claim 1 , wherein the request is one selected from a group consisting of:
a read request;
an update request;
an information request; and
a delete request.
4. The method of claim 1 , the method further comprising:
prior to obtaining the request:
obtaining, by a data processor, system metadata from the data nodes of the data nodes;
selecting, based on the system metadata, a second data node of the data nodes to perform DAG operations;
selecting, based on the system metadata, the data nodes of the data nodes to perform bidding counter operations;
selecting, based on the system metadata, the data nodes of the data nodes to perform metadata mapping operations; and
initiating the performance of:
a DAG on the second data node,
the bidding counter operations on each of the data nodes, and
the metadata mapping operation on each of the data nodes.
5. The method of claim 4 , wherein the system metadata specifies computational resources available to the data nodes.
6. A system for accessing data nodes of a data cluster, comprising:
a processor;
a data access gateway (DAG), which when executed by the processor performs a method, the method comprising:
obtaining, by the DAG, a request from a host; and
in response to the request:
obtaining bidding counters from data nodes, wherein the bidding counters comprise numerical representations of computational, network, and storage resources of the data nodes available to service the request;
obtaining metadata mappings from the data nodes, wherein:
a metadata mapping of the metadata mappings comprises a data node identifier associated with a data node of the data nodes corresponding to the metadata mapping and a plurality of chunk metadata; and
a chunk metadata of the plurality of chunk metadata may specify:
a chunk identifier associated with a corresponding chunk,
whether the chunk is a data chunk or a parity chunk,
whether the chunk is stored in an accelerator pool or a non-accelerator pool,
a storage location of the chunk, and
a data slice identifier associated with the chunk;
identifying, based on the bidding counters and metadata mappings, a data node of the data nodes associated with a highest bidding counter of the bidding counters and an appropriate metadata mapping of the metadata mappings, wherein the data node is associated with a highest amount of available computational, network, and storage resources, as specified by the highest bidding counter, compared to other data nodes of the data nodes; and
sending the request to the data node.
7. The system of claim 6 , wherein a bidding counter of the bidding counters is associated with a data node.
8. The system of claim 6 , wherein the request is one selected from a group consisting of:
a read request;
an update request;
an information request; and
a delete request.
9. The system of claim 6 , wherein the method further comprising:
prior to obtaining the request:
obtaining, by a data processor, system metadata from the data nodes of the data nodes;
selecting, based on the system metadata, a second data node of the data nodes to perform DAG operations;
selecting, based on the system metadata, the data nodes of the data nodes to perform bidding counter operations;
selecting, based on the system metadata, the data nodes of the data nodes to perform metadata mapping operations; and
initiating the performance of:
a DAG on the second data node,
the bidding counter operations on each of the data nodes, and
the metadata mapping operation on each of the data nodes.
10. The system of claim 9 , wherein the system metadata specifies computational resources available to the data nodes.
11. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for accessing data nodes of a data cluster, the method comprising:
obtaining, by a data access gateway (DAG), a request from a host; and
in response to the request:
obtaining bidding counters from data nodes, wherein the bidding counters comprise numerical representations of computational, network, and storage resources of the data nodes available to service the request;
obtaining metadata mappings from the data nodes, wherein:
a metadata mapping of the metadata mappings comprises a data node identifier associated with a data node of the data nodes corresponding to the metadata mapping and a plurality of chunk metadata; and
a chunk metadata of the plurality of chunk metadata may specify:
a chunk identifier associated with a corresponding chunk,
whether the chunk is a data chunk or a parity chunk,
whether the chunk is stored in an accelerator pool or a non-accelerator pool,
a storage location of the chunk, and
a data slice identifier associated with the chunk;
identifying, based on the bidding counters and metadata mappings, a data node of the data nodes associated with a highest bidding counter of the bidding counters and an appropriate metadata mapping of the metadata mappings, wherein the data node is associated with a highest amount of available computational, network, and storage resources, as specified by the highest bidding counter, compared to other data nodes of the data nodes; and
sending the request to the data node.
12. The non-transitory computer readable medium of claim 11 , wherein a bidding counter of the bidding counters is associated with a data node.
13. The non-transitory computer readable medium of claim 11 , wherein the request is one selected from a group consisting of:
a read request;
an update request;
an information request; and
a delete request.
14. The non-transitory computer readable medium of claim 11 , wherein the method further comprising:
prior to obtaining the request:
obtaining, by a data processor, system metadata from the data nodes of the data nodes;
selecting, based on the system metadata, a second data node of the data nodes to perform DAG operations;
selecting, based on the system metadata, the data nodes of the data nodes to perform bidding counter operations;
selecting, based on the system metadata, the data nodes of the data nodes to perform metadata mapping operations; and
initiating the performance of:
a DAG on the second data node,
the bidding counter operations on each of the data nodes, and
the metadata mapping operation on each of the data nodes.