Distributed computation system for servicing queries using revisions maps
In general, the invention relates to a method for servicing query requests. The method includes determining, by a first query node and in response to a first query request, a user data sub-region and a target sub-region associated with the first query request. The further includes identifying a first input sub-region of the target sub-region using a data flow graph, identifying a second query node associated with the first input sub-region, and issuing a second query request to the second query node to obtain data associated with the first input sub-region. The method further includes receiving, in response to the second query request, a second query result from the second query node, generating a first query result for the target sub-region using at least the second query result, and providing the first query result to an entity associated with the issuance of the first query request.
1. A method for servicing query requests, comprising:
determining, by a first query node and in response to a first query request, a user data sub-region and a target sub-region associated with the first query request,
wherein determining the user data sub-region associated with the query request comprising using a data flow graph,
wherein the data flow graph is associated with a data model and specifies relationships between a plurality of sub-regions in the data model, and
wherein the user data sub-region is one of the plurality of sub-regions;
issuing a revisions map request to a database management system;
obtaining, in response to the revisions map request, a revisions map, wherein the revisions map specifies a version of user data associated with the user data sub-region to use to service the query request, wherein the version of user data is one of a plurality of versions of user data;
identifying, using the data flow graph, a first input sub-region of the target sub-region;
identifying a second query node associated with the first input sub-region, wherein the second query node is a physical computing device or a logical computing device;
issuing a second query request to the second query node to obtain data associated with the first input sub-region;
receiving, by the second query node, the second query request, wherein the second query request comprises the revisions map;
making a first determination that data stored in a cache on the second query node is associated with the first input sub-region and is not associated with the version of the user data specified in the revisions man;
based on the first determination, making a second determination that the data associated with the first input sub-region is the user data;
based on the second determination, obtaining the user data associated with the first input sub-region from a persistent storage;
sending the second query result to the first query node, wherein the second query result comprises the user data obtained from the persistent storage;
receiving, in response to the second query request, a second query result from the second query node, wherein the second query result is based on the version of the user data;
generating a first query result for the target sub-region using at least the second query result; and
providing the first query result to an entity associated with the issuance of the first query request.
2. The method of claim 1 , further comprising:
receiving, by the second query node, the second query request, wherein the second query request comprises the revisions map;
determining that data stored in a cache on the second query node is associated with the first input sub-region and with the version of the user data specified in the revisions map; and
based on the determination, sending the second query result to the first query node, wherein the second query result comprises at least a portion of the data stored in the cache.
3. The method of claim 2 , wherein the data stored in the cache is derived data.
4. The method of claim 1 , further comprising:
receiving, by the second query node, the second query request;
identifying, using the data flow graph, a second input sub-region of the first input sub-region;
identifying a third query node associated with the second input sub-region;
issuing a third query request to the third query node to obtain data associated with the second input sub-region;
receiving a third query response from the third query node;
determining a value for at least one cell in the first input sub-region using the third query response; and
sending the second query result to the first query node, wherein the second query result comprises the value.
5. The method of claim 1 , wherein the version of user data is a most recent version of user data.
6. The method of claim 1 , wherein identifying the second query node associated with the first input sub-region comprises:
determining a distribution key using an outer dimension associated with the first input sub-region;
identifying the second query node using the distribution key and a node distribution key list.
7. The method of claim 6 , wherein the distribution key is a value generated using consistent hashing.
8. The method of claim 6 , wherein the second query node is a lead query node and wherein the second query node distributes updates to the node distribution key list to the first query node.
9. The method of claim 1 , wherein the entity is one selected from a group consisting of a client and a subscription engine.
10. The method of claim 1 , wherein the first query request is issued in response to a subscription engine receiving a notification that user data associated with the target sub-region has changed.
11. The method of claim 1 , wherein the data flow graph comprises at least one cycle.
12. The method of claim 1 , wherein the first query node comprises the target sub-region and a second sub-region, wherein each cell in the target sub-region and the second sub-region are associated with the same outer dimension.
13. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for servicing query requests, the method comprising:
determining, by a first query node and in response to a first query request, a user data sub-region and a target sub-region associated with the first query request,
wherein determining the user data sub-region associated with the query request comprising using a data flow graph,
wherein the data flow graph is associated with a data model and specifies relationships between a plurality of sub-regions in the data model, and
wherein the user data sub-region is one of the plurality of sub-regions;
issuing a revisions map request to a database management system;
obtaining, in response to the revisions map request, a revisions map, wherein the revisions map specifies a version of user data associated with the user data sub-region to use to service the query request, wherein the version of user data is one of a plurality of versions of user data;
identifying, using the data flow graph, a first input sub-region of the target sub-region;
identifying a second query node associated with the first input sub-region;
issuing a second query request to the second query node to obtain data associated with the first input sub-region, wherein the second query node is a physical computing device or a logical computing device;
receiving, by the second query node, the second query request, wherein the second query request comprises the revisions map;
making a first determination that data stored in a cache on the second query node is associated with the first input sub-region and is not associated with the version of the user data specified in the revisions map;
based on the first determination, making a second determination that the data associated with the first input sub-region is the user data;
based on the second determination, obtaining the user data associated with the first input sub-region from a persistent storage;
sending the second query result to a first query node, wherein the second query result comprises the user data obtained from the persistent storage;
receiving, in response to the second query request, a second query result from the second query node, wherein the second query result is based on the version of the user data;
generating a first query result for the target sub-region using at least the second query result; and
providing the first query result to an entity associated with the issuance of the first query request.
14. The non-transitory computer readable medium of claim 13 , wherein the method further comprises:
receiving, by the second query node, the second query request, wherein the second query request comprises the revisions map;
determining that data stored in a cache on the second query node is associated with the first input sub-region and with the version of the user data specified in the revisions map; and
based on the determination, sending the second query result to the first query node, wherein the second query result comprises at least a portion of the data stored in the cache.
15. The non-transitory computer readable medium of claim 13 , wherein identifying the second query node associated with the first input sub-region comprises:
determining a distribution key using an outer dimension associated with the first input sub-region; and
identifying the second query node using the distribution key and a node distribution key list,
wherein the distribution key is a value generated using consistent hashing.