Maximally distributed minimally coordinated systems and methods

1. A method for performing one or more database operations, the method comprising:

receiving, at a transaction manager, a client request to perform the one or more database operations with client data, the transaction manager in communication with a database coordinator having a transactional datastore that stores a metadata structure, and a system datastore including the client data;

sending the client request to the database coordinator to obtain a single-use storage identifier based on the client request, wherein the single-use storage identifier is stored in an internal non-durable memory;

receiving, at the transaction manager, the single-use storage identifier from the database coordinator, wherein the metadata structure is configured as a distributed Write-Ahead Log (WAL) to store the single-use storage identifier, and wherein the single-use storage identifier is specified in the metadata structure as a pending storage identifier;

storing, at the database coordinator, the pending storage identifier in temporary storage;

sending, from the transaction manager in receipt of the single-use storage identifier, an operation request to the system datastore, the operation request comprising instructions for the system datastore to perform the one or more database operations using the single-use storage identifier including a write operation for writing the client data to the system datastore; and

receiving an indication of whether the operation request was successfully performed.

2. The method of claim 1 , wherein the system datastore comprises a non-transactional datastore.

3. The method of claim 1 , further comprising, in accordance with receiving an indication that the write operation was successful, sending a finalization request to the database coordinator to store the single-use storage identifier in permanent storage.

4. The method of claim 1 , further comprising, in accordance with receiving an indication that the write operation was unsuccessful, sending a discard request to the database coordinator to discard the single-use storage identifier.

5. The method of claim 1 , wherein the operation request also comprises a read operation such that performing the one or more database operations comprises retrieving the client data from the system datastore, wherein obtaining the single-use storage identifier comprises identifying, by the database coordinator, the single-use storage identifier based on the client request.

6. The method of claim 5 , wherein receiving the indication of whether the one or more database operations was successfully completed comprises receiving the client data retrieved from the system datastore.

7. The method of claim 5 , further comprising processing, at the transaction manager, the client data retrieved from the system datastore.

8. The method of claim 1 , wherein the operation request comprises a deletion operation such that performing the one or more database operations comprises deleting the client data from the system datastore.

9. The method of claim 8 , wherein obtaining the single-use storage identifier comprises identifying, by the database coordinator, one or more discarded storage identifiers, the method further comprising, in accordance with receiving an indication that the delete operation was successful, sending a garbage collection request to the database coordinator to delete the one or more discarded storage identifiers corresponding to the delete operation.

10. The method of claim 1 , wherein the system datastore is configured to perform a plurality of database operations in parallel.

11. A datastore system, the datastore system comprising:

a memory; and

one or more hardware processors configured to perform a method comprising:

receiving, at a transaction manager, a client request to perform one or more database operations with client data, the transaction manager in communication with a database coordinator having a transactional datastore that stores a metadata structure, and a system datastore including the client data;

sending the client request to the database coordinator to obtain a single-use storage identifier based on the client request, wherein the single-use storage identifier is stored in an internal non-durable memory;

receiving, at the transaction manager, the single-use storage identifier from the database coordinator, wherein the metadata structure is configured as a distributed Write-Ahead Log (WAL) to store the single-use storage identifier, and wherein the single-use storage identifier is specified in the metadata structure as a pending storage identifier;

storing, at the database coordinator, the pending storage identifier in temporary storage;

sending, from the transaction manager in receipt of the single-use storage identifier, an operation request to the system datastore, the operation request comprising instructions for the system datastore to perform the one or more database operations using the single-use storage identifier including a write operation for writing the client data to the system datastore; and

receiving an indication of whether the operation request was successfully performed.

12. The datastore system of claim 11 , wherein the system datastore comprises a non-transactional datastore.

13. The datastore system of claim 11 , wherein the database coordinator does not directly communicate with the system datastore.

14. The datastore system of claim 11 , further comprising, in accordance with receiving an indication that the write operation was successful, sending a finalization request to the database coordinator to store the single-use storage identifier in permanent storage.

15. The datastore system of claim 11 , further comprising, in accordance with receiving an indication that the write operation was unsuccessful, sending a discard request to the database coordinator to discard the single-use storage identifier.

16. The datastore system of claim 11 , wherein the operation request comprises a deletion operation such that performing the one or more database operations comprises deleting the client data from the system datastore.

17. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of one or more electronic devices, cause the electronic devices to perform a method comprising:

receiving, at a transaction manager, a client request to perform one or more database operations with client data, the transaction manager in communication with a database coordinator having a transactional datastore that stores a metadata structure, and a system datastore including the client data;

sending the client request to the database coordinator to obtain a single-use storage identifier based on the client request, wherein the single-use storage identifier is stored in an internal non-durable memory;

receiving, at the transaction manager, the single-use storage identifier from the database coordinator, wherein the metadata structure is configured as a distributed Write—Ahead Log (WAL) to store the single-use storage identifier, and wherein the single-use storage identifier is specified in the metadata structure as a pending storage identifier;

storing, at the database coordinator, the pending storage identifier in temporary storage;

sending, from the transaction manager in receipt of the single-use storage identifier, an operation request to the system datastore, the operation request comprising further instructions for the system datastore to perform the one or more database operations using the single-use storage identifier including a write operation for writing the client data to the system datastore; and

receiving an indication of whether the operation request was successfully performed.