Encrypting object index in a distributed storage environment

1. A distributed storage system implemented in a cloud computing system, the distributed storage system comprising:

one or more processors; and

a tangible machine readable storage medium storing machine readable instructions that, when executed by the one or more processors, configure the one or more processors to:

receive requests to store data in the distributed storage system;

authenticate and authorize the requests;

generate system metadata for storing the data in the distributed storage system, the system metadata comprising data index information related to data storage locations;

store the data at the data storage locations;

after accumulating system metadata related to a plurality of data storage requests, batch the system metadata related to the plurality of data storage requests;

compress the system metadata that was batched to form compressed system metadata;

encrypt the compressed system metadata to form encrypted compressed system metadata; and

store the encrypted compressed system metadata in the distributed storage system.

2. The distributed storage system of claim 1 wherein the machine readable instructions configure the processor to:

compress at least a portion of the data to form a compressed portion of the data when compressing the system metadata;

encrypt the compressed portion of the data to form an encrypted compressed portion of the data when encrypting the compressed system metadata; and

store the encrypted compressed portion of the data with the encrypted compressed system metadata in the distributed storage system.

3. The distributed storage system of claim 1 wherein the data includes different types of data and wherein the system metadata includes different system metadata for each type of data.

4. The distributed storage system of claim 1 wherein the data is received at a first layer of servers, and wherein the data includes data from different sources having different identities that the first layer of servers is configured to authenticate and authorize.

5. The distributed storage system of claim 1 wherein the data includes different types of data from different sources having different identities, and wherein the machine readable instructions configure the processor to encrypt the system metadata generated for storing the data of the different types from the different sources using a single key.

6. The distributed storage system of claim 1 wherein the data includes different types of data from different sources having different identities, and wherein the machine readable instructions configure the processor to encrypt the system metadata generated for storing the data from the different sources using a separate key for each type of data.

7. The distributed storage system of claim 1 wherein the machine readable instructions configure the processor to encrypt each data unit of the compressed system metadata using a separate seed randomly generated for each data unit so that consecutive encrypted data units appear random.

8. The distributed storage system of claim 7 wherein the machine readable instructions configure the processor to store a seed used to encrypt a data unit of the compressed system metadata in unencrypted form in a header associated with the data unit encrypted and to use the seed stored in the header associated with the data unit encrypted when decrypting the data unit encrypted.

9. The distributed storage system of claim 1 wherein the machine readable instructions configure the processor to:

encrypt a first key used to encrypt the compressed system metadata using a second key associated with a cluster of servers; and

rotate the second key.

10. The distributed storage system of claim 1 wherein the machine readable instructions configure the processor to:

defragment data units storing unencrypted system metadata and associated data in the distributed storage system; and

compress and encrypt the unencrypted system metadata and associated data while defragmenting the data units.

11. The distributed storage system of claim 1 wherein the machine readable instructions configure the processor to:

select data units storing unencrypted system metadata and associated data in the distributed storage system based on at least one of an age and an amount of the unencrypted system metadata and associated data stored in the data units;

determine which of the data units selected include valid data based on a predetermined criterion, the valid data including valid unencrypted system metadata and associated data;

defragment the data units with valid data by writing the valid data from the data units to one or more new data units in the distributed storage system and by releasing the data units for storing new data;

compress and encrypt the valid data when writing the valid data to the one or more new data units; and

update the system metadata to point to the one or more new data units.

12. A method comprising:

accumulating, in first data structures in a first storage medium in a distributed storage system, data and system metadata generated for storing the data in the distributed storage system, the data received from a plurality of sources having distinct identities, and the data being of different types;

converting the first data structures, after a predetermined criterion is satisfied, into second data structures for storing in a second storage medium in the distributed storage system;

compressing and encrypting the data when converting the first data structures into the second data structures;

batching accumulated system metadata, and compressing and encrypting the accumulated system metadata to form compressed and encrypted system metadata; and

storing the second data structures including compressed and encrypted system metadata and data in the second storage medium in the distributed storage system.

13. The method of claim 12 , wherein:

encrypting the system metadata generated for storing the data of the different types from the plurality of sources comprises using a single key; or

encrypting the system metadata generated for storing the data from the plurality of sources comprises using a separate key for each type of data.

14. The method of claim 12 further comprising:

encrypting each data unit of compressed system metadata using a separate seed randomly generated for each data unit so that consecutive encrypted data units appear random;

storing a seed used to encrypt a data unit of the compressed system metadata in unencrypted form in a header associated with the data unit encrypted; and

using the seed stored in the header associated with the data unit encrypted when decrypting the data unit encrypted.

15. The method of claim 12 further comprising:

defragmenting data units storing unencrypted system metadata and associated data in the distributed storage system; and

compressing and encrypting the unencrypted system metadata and associated data while defragmenting the data units.

16. A distributed storage system comprising:

one or more processors; and

a tangible machine readable medium storing machine readable instructions that are executable by the one or more processors to:

receive data of one or more types from one or more sources having distinct identities for storing the data in a distributed storage system;

generate system metadata for storing the data in the distributed storage system;

store the system metadata generated for storing the data in data structures in the distributed storage system;

select a portion of the system metadata generated for storing the data in the distributed storage system;

batch the portion of the system metadata selected;

compress the portion of the system metadata that was batched to form compressed system metadata;

encrypt the compressed system metadata using a single key or using a separate key for each data type to form encrypted compressed system metadata; and

store the encrypted compressed system metadata in the distributed storage system.