Data management platform
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributed data management. One of the methods includes maintaining, by a first member in a distributed data management system having multiple computing members installed on multiple respective computers, a first garbage collection version vector that includes, for each member in the distributed data management system, a garbage collection version that represents a number of garbage collection processes performed by the member on a respective copy of a replicated data region maintained by the member in the data management system. If the first garbage collection version vector is different than a second garbage collection version vector received from a different provider member, a first replication process is performed that is different than a second replication process that is performed when the first garbage collection version vector matches the second garbage collection version vector.
1. A computer-implemented method for a distributed data management system to interact with a user, wherein the data management system has multiple computing members installed on multiple respective computers, and at least two types of data storage: nonvolatile, non-operational memory that stores persisted data regions, and operational, volatile memory that stores operational data, the method comprising:
receiving, from a user, a user-provided definition of a data region that defines a namespace of key-value pairs of data entries to be stored in the data region, wherein the user-provided definition includes one or more user-defined attributes for data entries in the data region;
receiving, from the user, a runtime query language statement defining one or more eviction criteria that specify one or more conditions under which data entries having particular values of one or more user-defined attributes should be evicted from operational memory of each computing member of the distributed data management system;
storing a portion of the data region in the operational memory of the multiple computing members;
initializing a respective eviction task on each member of the multiple computing members of the data management system;
repeatedly evaluating, by each respective eviction task on each member of the multiple computing members, the one or more eviction criteria over values of the one or more user-defined attributes of the one or more data entries of the data region stored in operational memory of the respective member;
determining, by a particular member of the multiple computing members, that the one or more eviction criteria are satisfied for a data entry of the data region stored in operational memory of the particular member; and
in response to determining that the one or more eviction criteria are satisfied for the data entry of the data region stored in operational memory of the particular member, removing the data entry of the data region from the operational memory of the particular member.
2. The method of claim 1 , further comprising:
in response to removing the data entry of the data region from the operational memory of the particular member, loading one or more data entries of a different second data region into the operational memory of the particular member.
3. The method of claim 1 , wherein removing the one or more data entries of the data region from the operational memory of the particular member comprises removing the one or more data entries of the data region from the operational memory of the particular member without propagating the removal of the one or more data entries to the non-operational memory of the particular member.
4. The method of claim 1 , wherein the operational memory has a lower access latency than the non-operational memory.
5. The method of claim 1 , wherein the one or more eviction criteria are specified by an eviction predicate, and wherein determining that the one or more eviction criteria are satisfied for a data entry of the data region stored in operational memory of the particular member comprises evaluating the eviction predicate on values of the one or more user-defined attributes of the data entry.
6. The method of claim 1 , wherein the one or more eviction criteria specify an eviction frequency, and further comprising evaluating the one or more eviction criteria for the one or more data entries of the data region according to the eviction frequency.
7. The method of claim 1 , wherein the one or more eviction criteria are specified by a structured query language statement or an object query language statement.
8. The method of claim 1 , further comprising:
storing the full data region in the non-operational memory of the distributed data management system.
9. A distributed data management system comprising multiple storage devices and multiple computing members installed on multiple respective computers, and at least two types of data storage: nonvolatile, non-operational memory that stores persisted data regions, and operational, volatile memory that stores operational data, the storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving, from a user, a user-provided definition of a data region that defines a namespace of key-value pairs of data entries to be stored in the data region, wherein the user-provided definition includes one or more user-defined attributes for data entries in the data region;
receiving, from the user, a runtime query language statement defining one or more eviction criteria that specify one or more conditions under which data entries having particular values of one or more user-defined attributes should be evicted from operational memory of each computing member of the distributed data management system;
storing a portion of the data region in the operational memory of the multiple computing members;
initializing a respective eviction task on each member of the multiple computing members of the data management system;
repeatedly evaluating, by each respective eviction task on each member of the multiple computing members, the one or more eviction criteria over values of the one or more user-defined attributes of the one or more data entries of the data region stored in operational memory of the respective member;
determining, by a particular member of the multiple computing members, that the one or more eviction criteria are satisfied for a data entry of the data region stored in operational memory of the particular member; and
in response to determining that the one or more eviction criteria are satisfied for the data entry of the data region stored in operational memory of the particular member, removing the data entry of the data region from the operational memory of the particular member.
10. The system of claim 9 , wherein the operations further comprise:
in response to removing the data entry of the data region from the operational memory of the particular member, loading one or more data entries of a different second data region into the operational memory of the particular member.
11. The system of claim 9 , wherein removing the one or more data entries of the data region from the operational memory of the particular member comprises removing the one or more data entries of the data region from the operational memory of the particular member without propagating the removal of the one or more data entries to the non-operational memory of the particular member.
12. The system of claim 9 , wherein the operational memory has a lower access latency than the non-operational memory.
13. The system of claim 9 , wherein the one or more eviction criteria are specified by an eviction predicate, and wherein determining that the one or more eviction criteria are satisfied for a data entry of the data region stored in operational memory of the particular member comprises evaluating the eviction predicate on values of the one or more user-defined attributes of the data entry.
14. The system of claim 9 , wherein the one or more eviction criteria specify an eviction frequency, and further comprising evaluating the one or more eviction criteria for the one or more data entries of the data region according to the eviction frequency.
15. The system of claim 9 , wherein the one or more eviction criteria are specified by a structured query language statement or an object query language statement.
16. The system of claim 9 , wherein the operations further comprise:
storing the full data region in the non-operational memory of the distributed data management system.
17. A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
at a distributed data management system comprising multiple computing members installed on multiple respective computers, and at least two types of data storage: non volatile, non-operational memory that stores persisted data regions, and operational, volatile memory that stores operational data, receiving, from a user, a user-provided definition of a data region that defines a namespace of key-value pairs of data entries to be stored in the data region, wherein the user-provided definition includes one or more user-defined attributes for data entries in the data region;
receiving, from the user, a runtime query language statement defining one or more eviction criteria that specify one or more conditions under which data entries having particular values of one or more user-defined attributes should be evicted from operational memory of each computing member of the distributed data management system;
storing a portion of the data region in the operational memory of the multiple computing members;
initializing a respective eviction task on each member of the multiple computing members of the data management system;
repeatedly evaluating, by each respective eviction task on each member of the multiple computing members, the one or more eviction criteria over values of the one or more user-defined attributes of the one or more data entries of the data region stored in operational memory of the respective member;
determining, by a particular member of the multiple computing members, that the one or more eviction criteria are satisfied for a data entry of the data region stored in operational memory of the particular member; and
in response to determining that the one or more eviction criteria are satisfied for the data entry of the data region stored in operational memory of the particular member, removing the data entry of the data region from the operational memory of the particular member.