Cross-file differential content synchronization
Methods, systems and techniques for synchronizing content, such as a file, between one or more clients and one or more servers are provided. Example embodiments provide a Cross-File Differential Content Synchronization System (CDCSS) to synchronize files between remote systems in a nearly instantaneous manner without necessitating the transfer of the entire contents of a file. These components cooperate to transfer only the differences in data content wherever possible to a recipient system by providing a patch mechanism that instructs the recipient to build the modified or new data content based upon content it already has access to. In addition, in one embodiment the CDCSS synchronization solution provides a client-based index, rather than a server-based index, for tracking patches and files, which reduces the amount of computational and I/O overhead on the server and which does not require a negotiation between a client and a server to provide an update.
1. A method for synchronizing a target file located on a server between the server and a client by generating the target file on the client, comprising:
conveying a patch for generating the target file on the client to the client from the server, the patch comprising a plurality of segments, each segment corresponding to one of a plurality of portions of the target file, wherein the entire target file is divided among the plurality of portions of the target file, each segment containing either data content of the corresponding portion or a reference to the data content of the corresponding portion, the reference comprising information for locating the referenced data content in files stored on the client without including the referenced data content;
conveying a manifest to the client, the manifest comprising a list that identifies the files referred to in segments of the patch containing references to the data content, for enabling the client to verify that the files referred to in the segments of the patch containing the references to the data content are available on the client;
generating the target file on the client by
retrieving the data content of portions of the target file from segments of the patch containing the data content of the corresponding portions; and
retrieving the data content of portions of the target file for which the reference to the data content of the corresponding portion is provided in the patch from the corresponding referenced files on the client.
2. The method of claim 1 , further comprising:
receiving a request at the server from the client for a missing file listed in the manifest in response to the client not being able to locate the missing file; and
in response, conveying the target file from the server to the client.
3. The method of claim 1 , further comprising:
receiving a request at the server from the client for the referenced data content of a missing file listed in the manifest in response to the client not being able to locate the missing file; and
in response, conveying the referenced data content from the server to the client.
4. The method of claim 1 , wherein the generating the target file comprises:
appending the data content from segments containing the data content to the generated file; and
locating the data content referenced by segments containing the references to data content and appending the located data content to the generated file.
5. The method of claim 1 , wherein the server receives the patch from a second client device.
6. The method of claim 1 , wherein the patch contains at least one segment that contains the reference to data content in a first file and at least one segment that contains the reference to data content in a second file, the second file being distinct and separate from the first file.
7. The method of claim 1 , wherein the entire target file is divided among the plurality of portions of the target file based on a chunking algorithm.
8. A non-transitory computer readable storage medium for synchronizing a target file located on a server between the server and a client by generating the target file on the client, comprising one or more sequences of instructions, the instructions when executed by one or more processors causing the one or more processors to execute the operations of:
conveying a patch for generating the target file on the client to the client from the server, the patch comprising a plurality of segments, each segment corresponding to one of a plurality of portions of the target file, wherein the entire target file is divided among the plurality of portions of the target file, each segment containing either data content of the corresponding portion or a reference to the data content of the corresponding portion, the reference comprising information for locating the referenced data content in files stored on the client without including the referenced data content;
conveying a manifest to the client, the manifest comprising a list that identifies the files referred to in segments of the patch containing references to the data content, for enabling the client to verify that the files referred to in the segments of the patch containing the references to the data content are available on the client;
generating the target file on the client by
retrieving the data content of portions of the target file from segments of the patch containing the data content of the corresponding portions; and
retrieving the data content of portions of the target file for which the reference to the data content of the corresponding portion is provided in the patch from corresponding referenced files on the client.
9. The non-transitory computer readable storage medium of claim 8 , further comprising instructions that when executed by the one or more processors cause the one or more processors to execute the operations of:
receiving a request at the server from the client for a missing file listed in the manifest in response to the client not being able to locate the missing file; and
in response, conveying the target file from the server to the client.
10. The non-transitory computer readable storage medium of claim 8 , further comprising instructions that when executed by the one or more processors cause the one or more processors to execute the operations of:
receiving a request at the server from the client for the referenced data content of a missing file listed in the manifest in response to the client not being able to locate the missing file; and
in response, conveying the referenced data content from the server to the client.
11. The non-transitory computer readable storage medium of claim 8 , wherein the generating the target file comprises:
appending the data content from segments containing the data content to the generated file; and
locating the data content referenced by segments containing the references to data content and appending the located data content to the generated file.
12. The non-transitory computer readable storage medium of claim 8 , wherein the server receives the patch from a second client device.
13. The non-transitory computer readable storage medium of claim 8 , wherein the patch contains at least one segment that contains the reference to data content in a first file and at least one segment that contains the reference to data content in a second file, the second file being distinct and separate from the first file.
14. The non-transitory computer readable storage medium of claim 8 , wherein the entire target file is divided among the plurality of portions of the target file based on a chunking algorithm.
15. A computing device for synchronizing a target file between the server and a client by generating the target file on the client, comprising one or more sequences of instructions, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the computing device to perform the steps of:
conveying a patch for generating the target file on the client to the client from the server, the patch comprising a plurality of segments, each segment corresponding to one of a plurality of portions of the target file, wherein the entire target file is divided among the plurality of portions of the target file, each segment containing either data content of the corresponding portion or a reference to the data content of the corresponding portion, the reference comprising information for locating the referenced data content in files stored on the client without including the referenced data content;
conveying a manifest to the client, the manifest comprising a list that identifies the files referred to in segments of the patch containing references to the data content, for enabling the client to verify that the files referred to in the segments of the patch containing the references to the data content are available on the client;
generating the target file on the client by
retrieving the data content of portions of the target file from segments of the patch containing the data content of the corresponding portions; and
retrieving the data content of portions of the target file for which the reference to the data content of the corresponding portion is provided in the patch from corresponding referenced files on the client.
16. The computing device of claim 15 , wherein the memory further includes instructions that when executed by the at least one processor, cause the computing device to perform the steps of:
receiving a request at the server from the client for a missing file listed in the manifest in response to the client not being able to locate the missing file; and
in response, conveying the target file from the server to the client.
17. The computing device of claim 15 , wherein the memory further includes instructions that when executed by the at least one processor, cause the computing device to perform the steps of:
receiving a request at the server from the client for the referenced data content of a missing file listed in the manifest in response to the client not being able to locate the missing file; and
in response, conveying the referenced data content from the server to the client.
18. The computing device of claim 15 , wherein the generating the target file comprises:
appending the data content from segments containing the data content to the generated file; and
locating the data content referenced by segments containing the references to data content and appending the located data content to the generated file.
19. The computing device of claim 15 , wherein the server receives the patch from a second client device.
20. The computing device of claim 15 , wherein the patch contains at least one segment that contains the reference to data content in a first file and at least one segment that contains the reference to data content in a second file, the second file being distinct and separate from the first file.