Systems and methods for processing-based file distribution in an aggregation of cloud storage services
The present disclosure describes systems and methods for aggregation and management of cloud storage to leverage processing provided by a cloud provider with aggregation of cloud storage services. Files may be pipelined to the processing provider and then resynchronized to an alternate storage service, with processing or metadata retained. In one implementation, files may be provided for processing, retrieved, and then separately provided for storage. In another implementation, files may be simultaneously provided for processing and storage at different cloud storage providers. After processing, the metadata of the file may be retrieved, and the file deleted from storage of the processing provider. The metadata may be synchronized with the stored file, or may be retained in metadata storage and retrieved by the client device during synchronization or file retrieval operations.
1. A method for pipelined processing via an aggregation of cloud storage services, comprising:
identifying, by a synchronization client of a client device, a first file in storage of the client device;
selecting, by the synchronization client, a first cloud storage service responsive to processing provided by the first cloud storage service for a file type of the first file;
transmitting, by the synchronization client, the first file to the first cloud storage service, the first cloud storage service performing the processing of the first file to generate metadata for the first file;
retrieving, by the synchronization client, the metadata from the first cloud storage service;
selecting, by the synchronization client, a second cloud storage service responsive to a storage policy associated with the client device for the second cloud storage service;
transmitting, by the synchronization client, the first file to the second cloud storage service, the second cloud storage service storing the first file; and
transmitting, by the synchronization client to the first cloud storage service, a request to delete the first file,
wherein the first cloud storage service performs the processing of the first file and the second cloud storage service stores the first file.
2. The method of claim 1 , wherein identifying the first file further comprises identifying a new or modified file.
3. The method of claim 1 , wherein the metadata comprises geolocation metadata for the file.
4. The method of claim 1 , wherein the metadata comprises facial recognition tagging metadata for the file.
5. The method of claim 1 , wherein transmitting the first file to the first cloud storage service and second cloud storage service are performed prior to retrieving the metadata from the first cloud storage service.
6. The method of claim 1 , further comprising transmitting the metadata to the second cloud storage service for association with the first file.
7. The method of claim 1 , further comprising transmitting the metadata to a third cloud storage service.
8. The method of claim 1 , wherein transmitting the request to delete the first file is performed responsive to retrieval, by the synchronization client, of the metadata for the first file.
9. The method of claim 1 , further comprising maintaining, by the synchronization client, a queue of files for transmission to the first cloud storage service; and
for each file of the queue of files:
transmitting, by the synchronization client, said file to the first cloud storage service, the first cloud storage service generating new metadata for said file,
retrieving, by the synchronization client, the metadata for said file from the first cloud storage service, and
transmitting, by the synchronization client to the first cloud storage service, a request to delete said file.
10. A method for pipelined processing via an aggregation of cloud storage services, comprising:
receiving, by a storage manager from a client device, an identification of a first file stored at the client device and designated for synchronization;
selecting, by the storage manager, a first cloud storage provider responsive to processing provided by the first cloud storage provider for a file type of the first file;
transmitting, by the storage manager to the client device, an identification of the first cloud storage provider, receipt of the identification of the first cloud storage provider triggering the client device to transmit the first file to the first cloud storage provider, receipt of the first file triggering the first cloud storage provider to perform the processing of the first file to generate metadata for the first file;
retrieving, by the storage manager from the first cloud storage provider, the metadata for the first file;
selecting, by the storage manager, a second cloud storage provider responsive to a storage policy associated with the client device for the second cloud storage provider;
transmitting, by the storage manager to the client device, an identification of the second cloud storage provider, receipt of the identification of the second cloud storage provider triggering the client device to transmit the first file to the second cloud storage provider, the second cloud storage provider storing the first file; and
transmitting, by the storage manager to the first cloud storage provider, a request to delete the first file,
wherein the first cloud storage provider performs the processing of the first file and the second cloud storage provider stores the first file.
11. The method of claim 10 , further comprising transmitting, by the storage manager, the metadata to the client device.
12. The method of claim 10 , wherein transmitting the request to delete the first file further comprises transmitting the request, by the storage manager, via the client device.
13. A system for pipelined processing via an aggregation of cloud storage services, comprising:
a client device, comprising a storage device and a processor executing a synchronization client;
wherein the synchronization client is configured for:
identifying a first file in the storage device,
selecting a first cloud storage service responsive to processing provided by the first cloud storage service for a file type of the first file,
transmitting the first file to the first cloud storage service, the first cloud storage service performing the processing of the first file to generate metadata for the first file,
retrieving the metadata from the first cloud storage service,
selecting a second cloud storage service responsive to a storage policy associated with the client device for the second cloud storage service,
transmitting the first file to the second cloud storage service, the second cloud storage service storing the first file; and
transmitting, to the first cloud storage service, a request to delete the first file, and
wherein the first cloud storage service performs the processing of the first file and the second cloud storage service stores the first file.
14. The system of claim 13 , wherein the synchronization client is further configured for identifying a new or modified file.
15. The system of claim 13 , wherein the metadata comprises geolocation metadata for the file or facial recognition tagging metadata for the file.
16. The system of claim 13 , wherein the synchronization client is further configured for transmitting the first file to the first cloud storage service and second cloud storage service prior to retrieving the metadata from the first cloud storage service.
17. The system of claim 13 , wherein the synchronization client is further configured for transmitting the metadata to the second cloud storage service for association with the first file.
18. The system of claim 13 , wherein the synchronization client is further configured for transmitting the metadata to a third cloud storage service.
19. The system of claim 13 , wherein the synchronization client is further configured for transmitting the request to delete the first file responsive to retrieval, by the synchronization client, of the metadata for the first file.
20. The system of claim 13 , wherein the synchronization client is further configured for:
maintaining a queue of files for transmission to the first cloud storage service; and
for each file of the queue of files:
transmitting said file to the first cloud storage service, the first cloud storage service generating new metadata for said file,
retrieving the metadata for said file from the first cloud storage service, and
transmitting, by the synchronization client to the first cloud storage service, a request to delete said file.