Updating a file using sync directories
A method for updating and formatting a local file is described. An update file, having a newer version of the local file, is accessed from a server over a network and a list of new hash values corresponding to data blocks of the update file is retrieved. Divisions between the data blocks are located when a value of a rolling checksum reaches a particular value, the data blocks being of variable size. New hash values of the update file are compared with local hash values of the local file. For each of the new hash values not having a matching local hash value, the corresponding data block is downloaded from the server and written to a new file. For each of the new hash values having a matching local hash value, the corresponding block is copied from the local file to the new file.
1. A method comprising:
generating a sync directory for a file, wherein generating the sync directory comprises:
sequentially, for each data unit of the file:
reading a next data unit from the file; and
calculating a rolling checksum based on the value of the next data unit and a specified amount of prior data from one or more prior data units of the file;
based on the rolling checksum value calculated for each data unit, determining whether a specified block size has been reached;
for each instance in which a determination is made that the specified block size has been reached:
calculating a hash value for a current block, the current block extending from an end of a last block or the beginning of the file if there is no last block to the current data unit; and
writing an offset value and the calculated hash value to the sync directory for that block; and
in response to an update operation, using the sync directory for the file and a new sync directory for an updated version of the file to generate a new version of the file.
2. The method of claim 1 , wherein the specified block size is a range between a minimum block size and a maximum block size.
3. The method of claim 1 , wherein the rolling checksum for one or more data units is calculated by taking a previous rolling checksum value, adding the value of the next data unit, and subtracting the oldest value of the previous rolling checksum.
4. The method of claim 1 , wherein an update operation comprises:
downloading the new sync directory for the updated version of the file;
for each hash value in the new sync directory, comparing the hash value for a matching hash value in the sync directory;
in response to determining a matching hash value, the corresponding block of the file is written to the new version of the file; and
in response to determining a non-matching hash value, the corresponding block is downloaded from the updated version of the file and written to the new version of the file.
5. The method of claim 1 , wherein after generating the sync directory for the file, the sync directory is stored in a reserved area within the file.
6. The method of claim 1 , wherein the new sync directory is downloaded from a remote server computer, wherein downloading the new synch directory comprises:
identifying, from the file, a URL pointing to the updated version of the file; and
obtaining the second sync directory using the URL.
7. The method of claim 1 , further comprising:
receiving, while generating the new version of the file, a request to access the file; and
in response to the request, providing access to the existing file in response to the request.
8. The method of claim 1 , wherein the sync directory includes a global hash based on contents of all of the blocks of the file, and the new sync directory includes a new global hash based on contents of all the blocks of the updated version of the file, and wherein the update operation is executed in response to a determination that the global hash and the new global hash do not match.
9. A system comprising:
one or more computers including one or more processors and one or more storage devices storing instructions that, when executed by the one or more computers, cause the system to perform operations comprising:
generating a sync directory for a file, wherein generating the sync directory comprises:
sequentially, for each data unit of the file:
reading a next data unit from the file; and
calculating a rolling checksum based on the value of the next data unit and a specified amount of prior data from one or more prior data units of the file;
based on the rolling checksum value calculated for each data unit, determining whether a specified block size has been reached;
for each instance in which a determination is made that the specified block size has been reached:
calculating a hash value for a current block, the current block extending from an end of a last block or the beginning of the file if there is no last block to the current data unit; and
writing an offset value and the calculated hash value to the sync directory for that block; and
in response to an update operation, using the sync directory for the file and a new sync directory for an updated version of the file to generate a new version of the file.
10. The system of claim 9 , wherein the specified block size is a range between a minimum block size and a maximum block size.
11. The system of claim 9 , wherein the rolling checksum for one or more data units is calculated by taking a previous rolling checksum value, adding the value of the next data unit, and subtracting the oldest value of the previous rolling checksum.
12. The system of claim 9 , wherein an update operation comprises:
downloading the new sync directory for the updated version of the file;
for each hash value in the new sync directory, comparing the hash value for a matching hash value in the sync directory;
in response to determining a matching hash value, the corresponding block of the file is written to the new version of the file; and
in response to determining a non-matching hash value, the corresponding block is downloaded from the updated version of the file and written to the new version of the file.
13. The system of claim 9 , wherein after generating the sync directory for the file, the sync directory is stored in a reserved area within the file.
14. The system of claim 9 , wherein the new sync directory is downloaded from a remote server computer, wherein downloading the new synch directory comprises:
identifying, from the file, a URL pointing to the updated version of the file; and
obtaining the second sync directory using the URL.
15. The system of claim 9 , further comprising instructions that, when executed by the one or more computers, cause the system to perform operations comprising:
receiving, while generating the new version of the file, a request to access the file; and
in response to the request, providing access to the existing file in response to the request.
16. The system of claim 9 , wherein the sync directory includes a global hash based on contents of all of the blocks of the file, and the new sync directory includes a new global hash based on contents of all the blocks of the updated version of the file, and wherein the update operation is executed in response to a determination that the global hash and the new global hash do not match.
17. One or more non-transitory computer readable storage media containing one or more computer programs that when executed by a computer system perform operations comprising:
generating a sync directory for a file, wherein generating the sync directory comprises:
sequentially, for each data unit of the file:
reading a next data unit from the file; and
calculating a rolling checksum based on the value of the next data unit and a specified amount of prior data from one or more prior data units of the file;
based on the rolling checksum value calculated for each data unit, determining whether a specified block size has been reached;
for each instance in which a determination is made that the specified block size has been reached:
calculating a hash value for a current block, the current block extending from an end of a last block or the beginning of the file if there is no last block to the current data unit; and
writing an offset value and the calculated hash value to the sync directory for that block; and
in response to an update operation, using the sync directory for the file and a new sync directory for an updated version of the file to generate a new version of the file.
18. The non-transitory computer readable storage media of claim 17 , wherein the specified block size is a range between a minimum block size and a maximum block size.
19. The non-transitory computer readable storage media of claim 17 , wherein the rolling checksum for one or more data units is calculated by taking a previous rolling checksum value, adding the value of the next data unit, and subtracting the oldest value of the previous rolling checksum.
20. The non-transitory computer readable storage media of claim 17 , wherein an update operation comprises:
downloading the new sync directory for the updated version of the file;
for each hash value in the new sync directory, comparing the hash value for a matching hash value in the sync directory;
in response to determining a matching hash value, the corresponding block of the file is written to the new version of the file; and
in response to determining a non-matching hash value, the corresponding block is downloaded from the updated version of the file and written to the new version of the file.
21. The non-transitory computer readable storage media of claim 17 , wherein the new sync directory is downloaded from a remote server computer, wherein downloading the new synch directory comprises:
identifying, from the file, a URL pointing to the updated version of the file; and
obtaining the second sync directory using the URL.