Update and query of a large collection of files that represent a single dataset stored on a blob store

1. A system for updating data, comprising:

an interface configured to:

receive a table indication of a table, wherein the table comprises data and metadata, wherein the metadata includes a list of files, file partitioning information, and a transaction log comprising a plurality of transaction files, wherein each of the transaction files is associated with one or more files, wherein each of the plurality of transaction files is associated with a corresponding transaction log identifier; and

receive a transaction indication to perform a transaction; and

a processor configured to:

determine a current position N in the transaction log, wherein the current position N is associated with a first transaction file that has a first transaction log identifier, wherein N is an integer;

determine a current state of the metadata;

determine a read set associated with the transaction, wherein the read set associated with the transaction is associated with a first set of files;

attempt to write an update to the transaction log associated with a next position N+1, wherein the next position N+1 is associated with a second transaction file that has a second transaction log identifier, wherein the second transaction log identifier is a next sequence transaction log identifier after the first transaction log identifier;

in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists:

determine a set of updated files associated with the simultaneous transaction;

determine whether the set of updated files associated with the simultaneous transaction overlap with the first set of files included in the read set associated with the transaction; and

in response to a non-overlap determination that there is not an overlap between the read set associated with the transaction comprised of the first set of files and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction log associated with a further position N+2.

2. The system of claim 1 , wherein the system for updating data comprises a cluster computing system.

3. The system of claim 2 , wherein a plurality of cluster computing systems comprise systems for updating data on the table.

4. The system of claim 1 , wherein the table is scalable to hold large data sets.

5. The system of claim 4 , wherein large data sets comprise data sets holding gigabytes of data, terabytes of data, petabytes of data, or exabytes of data.

6. The system of claim 1 , wherein the metadata includes file statistical information, a transaction identifier, a column invariant, or a table schema.

7. The system of claim 6 , wherein file statistical information comprises column minima, column maxima, a count of rows, average data length per column, a count of null cells per column, a data histogram, a number of distinct values per column, or bloom filter data.

8. The system of claim 1 , wherein the processor is further configured to process the transaction to determine the update.

9. The system of claim 1 , wherein the processor is further configured to, in response to a no-transaction determination that no simultaneous transaction associated with the next position N+1 already exists, indicate transaction success.

10. The system of claim 1 , wherein the processor is further configured to, in response to an overlap determination that there is an overlap between the read set associated with the transaction and the set of updated files associated with the simultaneous transaction, indicate transaction failure.

11. The system of claim 1 , wherein determining the read set associated with the transaction additionally comprises determining that metadata was read by the transaction, determining a set of files read by the transaction, or determining a transaction identifier.

12. The system of claim 1 , wherein to determine the non-overlap determination, the processor is configured to determine whether a file of the set of updated files associated with the simultaneous transaction is identified by the read set associated with the transaction.

13. The system of claim 1 , wherein to determine the non-overlap determination, the processor is configured to perform a semantic comparison on the read set associated with the transaction and the set of updated files associated with the simultaneous transaction.

14. The system of claim 1 , wherein to determine the non-overlap determination, the processor is configured to determine that no files added by the simultaneous transaction overlap an element of the read set.

15. The system of claim 1 , wherein to determine the non-overlap determination, the processor is configured to determine that no files modified by the simultaneous transaction satisfy the read set.

16. The system of claim 1 , wherein to determine the non-overlap determination, the processor is configured to determine that no files deleted by the simultaneous transaction satisfy the read set.

17. The system of claim 1 , wherein to determine the non-overlap determination, the processor is configured to determine that metadata read by the transaction was not modified by the simultaneous transaction.

18. The system of claim 1 , wherein to determine the non-overlap determination, the processor is configured to determine that a transaction identifier associated with the transaction and a transaction identifier associated with the simultaneous transaction do not match.

19. The system of claim 1 , wherein the processor is further configured to determine whether the transaction determination that the simultaneous transaction associated with the next position N+1+ already exists, wherein the transaction determination that the simultaneous transaction associated with the next position N+1+ already exists is based at least in part on a table data store type.

20. The system of claim 1 , wherein the plurality of transaction files are update files, wherein the transaction log comprises an ordered list of the update files, wherein each update file is associated with a consecutively assigned position number.

21. The system of claim 1 , wherein to attempt to write the update to the transaction log associated with a further position N+2, the processor is configured to trigger a recursive look, and in response to a determination that the recursive look allows writing, the processor is configured to write the update to the transaction log.

22. A method for updating data, comprising:

receiving a table indication of a table, wherein the table comprises data and metadata, wherein the metadata includes a list of files, file partitioning information, and a transaction log comprising a plurality of transaction files, wherein each of the transaction files is associated with one or more files, wherein each of the plurality of transaction files is associated with a corresponding transaction log identifier;

receiving a transaction indication to perform a transaction;

determining a current position N in the transaction log, wherein the current position N is associated with a first transaction file that has a first transaction log identifier, wherein N is an integer;

determining a current state of the metadata;

determining a read set associated with the transaction, wherein the read set associated with the transaction is associated with a first set of files;

attempting to write an update to the transaction log associated with a next position N+1, wherein the next position N+1 is associated with a second transaction file that has a second transaction log identifier, wherein the second transaction log identifier is a next sequence transaction log identifier after the first transaction log identifier;

in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists:

determining a set of updated files associated with the simultaneous transaction;

determining whether the set of updated files associated with the simultaneous transaction overlap with the first set of files included in the read set associated with the transaction; and

in response to a non-overlap determination that there is not an overlap between the read set associated with the transaction comprised of the first set of files and the set of updated files associated with the simultaneous transaction, attempting to write the update to the transaction log associated with a further position N+2.

23. A computer program product for updating data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving a table indication of a table, wherein the table comprises data and metadata, wherein the metadata includes a list of files, file partitioning information, and a transaction log comprising a plurality of transaction files, wherein each of the transaction files is associated with one or more files, wherein each of the plurality of transaction files is associated with a corresponding transaction log identifier;

receiving a transaction indication to perform a transaction;

determining a current position N in the transaction log, wherein the current position N is associated with a first transaction file that has a first transaction log identifier, wherein N is an integer;

determining a current state of the metadata;

determining a read set associated with the transaction, wherein the read set associated with the transaction is associated with a first set of files;

attempting to write an update to the transaction log associated with a next position N+1, wherein the next position N+1 is associated with a second transaction file that has a second transaction log identifier, wherein the second transaction log identifier is a next sequence transaction log identifier after the first transaction log identifier;

in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists:

determining a set of updated files associated with the simultaneous transaction;

determining whether the set of updated files associated with the simultaneous transaction overlap with the first set of files included in the read set associated with the transaction; and

in response to a non-overlap determination that there is not an overlap between the read set associated with the transaction comprised of the first set of files and the set of updated files associated with the simultaneous transaction, attempting to write the update to the transaction log associated with a further position N+2.