System and method for direct object to file mapping in a global filesystem

1. A method for storing a file in cloud storage service (CSS) having a blocks index for indexing blocks that each have a unique block identifier, each block identifier being based on the content of the respective one of the blocks that it identifies, the blocks index having entries indicating for each block identifier at least one location of the block within an object storage system (OSS), comprising:

transmitting, from the CSS, a list of at least one block identifier indicating at least one respective block that is not in the blocks index but which is indicated by a received file map for the file, each block on the list having a unique block identifier;

adding, by the CSS, for each respective block indicated on the list and that has been successfully uploaded to the OSS, an entry into the blocks index to indicate a location of the uploaded block within the OSS; and

when all of the blocks indicated on the list have been successfully uploaded, concatenating all of the blocks of the received file map in an order specified by the received file map to form a file object corresponding to the file in the OSS;

wherein the transmitted list further includes at least one upload token, wherein the at least one upload token specifies information for a client device to successfully upload to the OSS one or more of the blocks that the upload token is for.

2. The method of claim 1 , at least one of the at least one successfully uploaded block is retained in the OSS and its respective corresponding entry is retained in the blocks index until being deleted by a blocks cleaner unit.

3. The method of claim 1 , wherein each respective block that has been successfully uploaded to the OSS is stored in a first bucket of the OSS and the file object is stored in a second bucket of the OSS.

4. The method of claim 1 , wherein the concatenation is performed by at least one or more of the: the CSS, the OSS, and a concatenation service.

5. The method of claim 1 , wherein the file object is stored with a unique identifier based on a name of the file and a path of the file.

6. The method of claim 1 , further comprising:

adding, for at least one unique block indicated by the received file map, an additional entry to the blocks index, the additional entry being at least a location in the OSS of the file object.

7. The method of claim 6 , wherein the additional entry further specifies a byte range within the file object.

8. The method of claim 6 , further comprising:

when a block identifier for a block indicated by the received file map corresponds to at least one entry in the blocks index that is a location in the OSS of a file object, calculating a block identifier for the indicated block based on the content of the file object indicated by the location;

when the block identifier calculated based on the content of the file object indicated by the location does not match the block identifier indicated by the received file map, indicating the occurrence of an integrity error; and

recovering from the integrity error.

9. The method of claim 8 , wherein recovering from the integrity error comprises finding in the OSS an additional location of the block indicated by the received file map for which a block identifier that is calculated based on the content indicated by the additional location matches the block identifier indicated by the received file map, wherein the additional location is one of: a deleted files trashcan, a previous file versions store, and a location indicated by an additional entry in the blocks index.

10. The method of claim 1 , further comprising:

adding, for at least one unique block indicated by the received file map, an additional entry to the blocks index, the additional entry being at least a location of a container object containing at least the at least one unique block indicated by the received file map and a different other at least one unique block indicated by the received file map.

11. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process for storing a file in a cloud storage service (CSS) having a blocks index for indexing blocks that each have a unique block identifier, each block identifier being based on the content of the respective one of the blocks that it identifies, the blocks index having entries indicating for each block identifier at least one location of the block within an object storage system (OSS), the process comprising:

transmitting, from the CSS, a list of at least one block identifier indicating at least one respective block that is not in the blocks index but which is indicated by a received file map for the file, each block on the list having a unique block identifier;

adding, by the CSS, for each respective block indicated on the list and that has been successfully uploaded to the OSS, an entry into the blocks index to indicate a location of the uploaded block within the OSS; and

when all of the blocks indicated on the list have been successfully uploaded, concatenating all of the blocks of the received file map in an order specified by the received file map to form a file object corresponding to the file in the OSS;

wherein the transmitted list further includes at least one upload token, wherein the at least one upload token specifies information for a client device to successfully upload to the OSS one or more of the blocks that the upload token is for.

12. A system for storing a file in cloud storage service (CSS) having a blocks index for indexing blocks that each have a unique block identifier, each block identifier being based on the content of the respective one of the blocks that it identifies, the blocks index having entries indicating for each block identifier at least one location of the block within an object storage system (OSS), comprising:

a processing circuitry; and

a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:

transmit, from the CSS, a list of at least one block identifier indicating at least one respective block that is not in the blocks index but which is indicated by a received file map for the file, each block on the list having a unique block identifier;

add, by the CSS, for each respective block indicated on the list and that has been successfully uploaded to the OSS, an entry into the blocks index to indicate a location of the uploaded block within the OSS; and

when all of the blocks indicated on the list have been successfully uploaded, concatenate all of the blocks of the received file map in an order specified by the received file map to form a file object corresponding to the file in the OSS;

wherein the transmitted list further includes at least one upload token, wherein the at least one upload token specifies information for a client device to successfully upload to the OSS one or more of the blocks that the upload token is for.

13. The system of claim 12 , wherein at least one of the at least one successfully uploaded block is retained in the OSS and its respective corresponding entry is retained in the blocks index until being deleted by a blocks cleaner unit.

14. The system of claim 12 , wherein the system is further configured to:

add, for at least one unique block indicated by the received file map, an additional entry to the blocks index, the additional entry being at least a location in the OSS of the file object.

15. The system of claim 14 , wherein the additional entry further specifies a byte range within the file object.

16. The system of claim 14 , wherein the system is further configured to:

when a block identifier for a block indicated by the received file map corresponds to at least one entry in the blocks index that is a location in the OSS of a file object, calculate a block identifier for the indicated block based on the content of the file object indicated by the location;

when the block identifier calculated based on the content of the file object indicated by the location does not match the block identifier indicated by the received file map, indicate the occurrence of an integrity error; and recover from the integrity error.

17. The system of claim 16 , wherein, to recover from the integrity error, wherein the system is further configured to:

finding in the OSS an additional location of the block indicated by the received file map for which a block identifier that is calculated based on the content indicated by the additional location matches the block identifier indicated by the received file map, wherein the additional location is one of: a deleted files trashcan, a previous file versions store, and a location indicated by an additional entry in the blocks index.

18. The system of claim 12 , wherein the system is further configured to:

add, for at least one unique block indicated by the received file map, an additional entry to the blocks index, the additional entry being at least a location of a container object containing at least the at least one unique block indicated by the received file map and a different other at least one unique block indicated by the received file map.