Method for aggregating distributed data
A customer of a computing resource service provider submits a request to a service in a first data region to obtain all customer data stored across all data regions managed by the computing resource service provider. Accordingly, the service may refer to a chatter table that includes a listing of locations wherein customer data may be stored to determine which data regions comprise customer data. Based on the determination, the service may transmit one or more requests to other services within the determined data regions to obtain the customer data while ignoring any data regions that do not comprise customer data. Upon receiving the customer data, the service may aggregate this data and provide the aggregated data to the customer to fulfill the customer request.
1. A computer-implemented method, comprising:
transmitting, from a first location to a second location, an update to a first data structure of the second location that indicates a first set of locations where subsets of data are stored to indicate that a first subset of the data is stored at the first location;
receiving, at the first location, a request to obtain the data;
determining, based at least in part on a second data structure of the first location that indicates a second set of locations where the subsets of the data are stored, that the second location stores a second subset of the data by at least locating a customer identification value within the second data structure to identify one or more identifiers for the second location that indicate locations of subsets of the data;
obtaining aggregated data by at least:
receiving, from the second location, the second subset of the data; and
aggregating the second subset of the data with the first subset of the data; and
transmitting the aggregated data in response to the request.
2. The computer-implemented method of claim 1 , wherein the first location comprises a plurality of computing devices that implement one or more services that access the second data structure to fulfill the request.
3. The computer-implemented method of claim 1 , wherein the second location comprises a plurality of computing devices that implement one or more services that:
obtain the second subset of the data in response to the request; and
transmit the second subset of the data.
4. The computer-implemented method of claim 1 , further comprising maintaining the first data structure and the second data structure to be consistent across the first and second locations.
5. The computer-implemented method of claim 4 , wherein maintaining the first data structure to be consistent includes transmitting, from the second location to the first location, an additional update to the second data structure of the first location.
6. The computer-implemented method of claim 1 , wherein obtaining the aggregated data is performed without requesting data from at least one location as a result of the at least one location lacking data associated with the request.
7. A computer system, comprising:
one or more processors; and
memory storing instructions that, if executed by the computer system, cause the computer system to:
store, at a first location, first data;
transmit, from the first location to a second location, an update to a first data structure associated with the second location to cause the first data structure to indicate the first data is stored at the first location;
receive a notification from the second location that second data is stored at the second location;
update a second data structure associated with the first location to indicate that the second location stores the second data, the first and second data structures comprising:
a plurality of identifiers for a plurality of locations that store data associated with a plurality of users of the computer system; and
a plurality of user identification values usable to obtain the plurality of identifiers for the plurality of locations that store the data associated with the plurality of users of the computer system; and
in response to a request for data, receive, a result of the second data structure associated with the first location indicating that the second location stores the second data, the second data from the second location and aggregating the second data with at least the first data.
8. The computer system of claim 7 , wherein the data structure associated with the first location further comprises an identifier associated with a user of the computer system for the first location.
9. The computer system of claim 7 , wherein the first location and the second location further each comprise a plurality of computing devices that implement one or more services that obtain data associated with a user in response to the request from the user to locate the data associated with the user across multiple locations.
10. The computer system of claim 7 , wherein the computer system is able to access, on behalf of a user, the first location and the second location to generate, manage, and obtain the data.
11. The computer system of claim 7 , wherein the first data structure associated with the second location is accessible to locate data associated with a user across the first and the second locations regardless of whether the second location comprises the data associated with the user.
12. The computer system of claim 7 , wherein the instructions, further in response to the request for data, cause the computer system to: submit an additional request to the second location for the second data; and receive the second data in response to the additional request.
13. The computer system of claim 12 , wherein the additional request in an application programming interface call.
14. A non-transitory computer-readable storage medium storing instructions that, if executed by one or more processors of a computer system, cause the computer system to at least:
transmit, from a first location to a second location, an update to a first data structure of the second location that indicates a first set of locations where subsets of data are stored to indicate that a first subset of the data is stored at the first location;
receive, at the first location, a request for the data;
identify, based at least in part on a second data structure of the first location that indicates a second set of locations where the subsets of the data are stored, other locations that store a second subset of the data by at least causing the computer system to locate a user identification value within the second data structure to obtain one or more identifiers for the other locations;
transmit, to each location of the other locations that store the second subset of the data, a request to obtain at least a portion of the second subset of the data; and
in response to receipt of the second subset of the data, transmit the first subset of the data and the second subset of the data.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions further cause the computer system to aggregate the first subset of the data and the second subset of the data such that the data is provided to fulfill the request.
16. The non-transitory computer-readable storage medium of claim 14 , wherein the first data structure and the second data structure are maintained to be consistent across the first location and the second location.
17. The non-transitory computer-readable storage medium of claim 14 , wherein receiving the second subset of the data is performed without requesting data from at least one location other than the other locations as a result of the at least one location lacking another portion of the second subset of the data.
18. The non-transitory computer-readable storage medium of claim 14 , wherein each location of the other locations comprises a plurality of computing devices that implement one or more services that obtain portions of the second subset of the data in response to the transmitted request and to transmit the portions of the second subset of the data.
19. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions further cause the computer system to access the first location to access the second data structure to fulfill the request.
20. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions that cause the computer system to receive the second subset of the data further cause the computer system to, from each location of the other locations: receive the portion requested from the location; and aggregate the portion with other portions from the other location to produce the second subset of the data.