Hosted file sync with stateless sync nodes
Disclosed herein is a system and method to synchronize files from a hosted file system where the client endpoint devices have direct access to the files stored on a hosted file store. When the client desires to synchronize with the files stored on the hosted file store the process identifies a sync role that can process the request. The particular sync role does not need to maintain state with the client device or the file store. The state necessary for the processing of the synchronization process is maintained separate from the sync role such that any sync role in the system can pick up the requests and process the requests. Once the role has been selected the synchronization process proceeds as normal. As state is not maintained by the sync role multiple roles can process the request or two roles can handle different portions of the process.
1. A method comprising:
synchronizing a set of files and/or folders between a client device and a remote file server, via a sync node, and storing a synchronization state of the synchronizing by the sync node within a runtime state store managed independent of the sync node by a host service;
receiving, by the host service from the client device, a sync request for synchronizing the set of files and/or folders between the client device and the remote file store;
selecting, by the service, one or more stateless sync nodes from among a plurality of stateless sync nodes of the host service, for resuming synchronizing the set of files and/or folders between the client device and the remote file server, wherein the one or more stateless sync nodes do not maintain a synchronization state between the client device and the remote file store for the set of files and/or folders;
transmitting, by the host service, the synchronization state of the set of files and/or folders of the synchronizing performed by the sync node from the runtime state store to the one or more stateless sync nodes;
continuing synchronizing, by the one or more stateless sync nodes, the set of files and/or folders based on the synchronization state provided by the host service from the runtime state store, where the continuing of the synchronizing results in one or more changes to the client device and/or the remote file store; and
committing, by the service, the one or more changes.
2. The method of claim 1 wherein the method further comprises determining whether the sync request can be processed based on whether the client device is authorized to access the file store.
3. The method of claim 1 , further comprising identifying, by the host service, a long-running sync task included in the continuing of the synchronizing, and offloading the identified long-running sync task from the one or more stateless sync nodes to a background node of the host service is-configured for executing the long running sync task.
4. The method of claim 1 further comprising sharding a task of the sync request when a size of the task exceeds a threshold size.
5. The method of claim 4 wherein the selecting comprises selecting a plurality of stateless sync nodes of the host service and distributing processing of the shards among the plurality of selected stateless sync nodes such that each selected stateless sync node processes at least a respective shard of the sync request.
6. The method of claim 1 wherein the one or more stateless sync nodes are selected from among the plurality of stateless sync nodes of the service based on a round robin approach.
7. The method of claim 1 wherein the one or more stateless sync nodes are selected based upon a usage level for each of the stateless sync nodes among the plurality of stateless sync nodes.
8. A computing device comprising:
memory that is coupled to at least one hardware computing device and that includes computer executable instructions that, based on execution by the at least one hardware computing device, configure a service to perform actions comprising:
synchronizing a set of files and/or folders between a client device and a remote file server, via a sync node, and storing a synchronization state of the synchronizing by the sync node within a runtime state store managed independent of the sync node by a host service;
receiving, by the host service from the client device, a sync request for synchronizing the set of files and/or folders between the client device and the remote file store;
selecting, by the host service, one or more stateless sync nodes from among a plurality of stateless sync nodes of the host service, for synchronizing the set of files and/or folders between the client device and the remote file server, wherein the one or more stateless sync nodes do not maintain a synchronization state between the client device and the remote file store for the set of files and/or folders;
transmitting, by the host service, the synchronization state of the set of files and/or folders of the synchronizing performed by the sync node from the runtime state store to the one or more stateless sync nodes;
continuing synchronizing, by the one or more stateless sync nodes, the set of files and/or folders based on the synchronization state provided by the host service from the runtime state store, where the continuing of the synchronizing results in one or more changes to the client device and/or the remote file store; and
committing, by the service, the one or more changes.
9. The computing device of claim 8 wherein the actions further comprise determining whether the sync request can be processed based on whether the client device is authorized to access the file store.
10. The computing device of claim 8 wherein the actions further comprises identifying, by the host service, a long-running sync task included in the continuing of the synchronizing, and offloading the identified long-running sync task from the one or more stateless sync nodes to a background node of the host service is configured for executing the long running sync task.
11. The computing device of claim 8 wherein a plurality of stateless sync nodes of the service are each configured to process at least a shard of the sync request.
12. The computing device of claim 8 wherein the one or more stateless sync nodes are selected from among the plurality of stateless sync nodes of the service based upon a usage level for each of the plurality of stateless sync nodes.
13. The computing device of claim 8 , wherein the actions further comprise sharding a task of the sync request into shards when a size of the task exceeds a threshold size, and distributing processing of the shards among a plurality of selected stateless sync nodes such that each selected stateless sync node processes at least a respective shard of the sync request.
14. A non-transitory computer-readable medium storing program instructions which when executed by a processor cause a computer to perform a method comprising:
synchronizing a set of files and/or folders between a client device and a remote file server, via a sync node, and storing a synchronization state of the synchronizing by the sync node within a runtime state store managed independent of the sync node by a host service;
receiving, by the host service from the client device, a sync request for synchronizing the set of files and/or folders between the client device and the remote file store;
selecting, by the host service, one or more stateless sync nodes from among a plurality of stateless sync nodes of the host service, for synchronizing the set of files and/or folders between the client device and the remote file server, wherein the one or more stateless sync nodes do not maintain a synchronization state between the client device and the remote file store for the set of files and/or folders;
transmitting, by the host service, the synchronization state of the set of files and/or folders of the synchronizing performed by the sync node from the runtime state store to the one or more stateless sync nodes;
continuing synchronizing, by the one or more stateless sync nodes, the set of files and/or folders based on the synchronization state provided by the host service from the runtime state store, where the continuing of the synchronizing results in one or more changes to the client device and/or the remote file store; and
committing, by the service, the one or more changes.
15. The non-transitory computer-readable medium of claim 14 where the method further comprises determining whether the sync request can be processed based on whether the client device is authorized to access the file store.
16. The non-transitory computer-readable medium of claim 14 wherein the method further comprises identifying, by the host service, a long-running sync task included in the continuing of the synchronizing, and offloading the identified long-running sync task from the one or more stateless sync nodes to a background node of the host service is-configured for executing the long running sync task.
17. The non-transitory computer-readable medium of claim 14 , wherein the actions further comprise sharding a task of the sync request when a size of the task exceeds a threshold size.
18. The non-transitory computer-readable medium of claim 17 where the selecting comprises selecting a plurality of stateless sync nodes of the host service and distributing processing of the shards among the plurality of selected stateless sync nodes such that each selected stateless sync node processes at least a respective shard of the task.
19. The method of claim 1 , wherein the one or more stateless sync nodes are selected based on a load among the plurality of stateless sync nodes.
20. The method of claim 1 , wherein the one or more stateless sync nodes are selected based on a type of task associated with the sync request.