Fully managed account level blob data encryption in a distributed storage environment

1. A system comprising:

a hardware processor; and

machine readable instructions, stored on a tangible machine readable device, when executed by the hardware processor, configure the hardware processor to:

receive a request to store data in a cluster in a cloud computing system, the request received at a first layer of servers in the cluster, the first layer of servers configured to authenticate and authorize the request;

compress the data at the first layer of servers upon authentication and authorization of the request;

encrypt the compressed data at the first layer of servers when encryption is enabled;

send the request and the encrypted data to a second layer of servers in the cluster, the second layer of servers configured to store data structures used to manage data storage in a third layer of servers in the cluster, and to distribute the request and the encrypted data to the third layer of servers using the data structures; and

store the encrypted data in the third layer of servers, wherein encrypting the data at the first layer of servers reduces latency experienced by an end user associated with transferring the data between the first, second, and third layers of servers.

2. The system of claim 1 wherein the machine readable instructions configure the hardware processor to:

allow dynamic enabling and disabling of the encryption; and

append the encrypted data to unencrypted data stored in the third layer of servers and append unencrypted data to the encrypted data stored in the third layer of servers depending on whether the encryption is enabled or disabled.

3. The system of claim 2 wherein the machine readable instructions configure the hardware processor to track encryption status of data stored in the third layer of servers to indicate whether the data stored in the third layer of servers is unencrypted, encrypted, or partially encrypted due to the dynamic enabling and disabling of the encryption.

4. The system of claim 1 wherein the machine readable instructions configure the hardware processor to:

compress, at the first layer of servers, metadata received with the data;

encrypt, at the first layer of servers, the compressed metadata received with the data when encryption is enabled; and

store the encrypted metadata in the third layer of servers.

5. The system of claim 1 wherein the machine readable instructions configure the hardware processor to encrypt, when encryption is enabled, each block of data using a unique key so that consecutive encrypted blocks appear random.

6. The system of claim 5 wherein the machine readable instructions configure the hardware processor to generate each unique key based on a unique seed randomly generated for each block and a first key assigned to an account to which the data belongs.

7. The system of claim 6 wherein the machine readable instructions configure the hardware processor to store a seed used to encrypt a block in unencrypted form in a header associated with the encrypted block and to use the seed stored in the header associated with the encrypted block when decrypting the encrypted block.

8. The system of claim 6 wherein the machine readable instructions configure the hardware processor to encrypt the first key using a second key associated with the cluster.

9. The system of claim 8 wherein the machine readable instructions configure the hardware processor to rotate the second key.

10. The system of claim 1 wherein the machine readable instructions configure the hardware processor to write, using the data structure, a record to a log for each received request to write data, and to reproduce, using the log, the request to write data when a server in the second layer of servers crashes.

11. The system of claim 1 wherein the machine readable instructions configure the hardware processor to store the encrypted data in data units in the third layer of servers, and to selectively move content of a first data unit to a second data unit to make the first data unit available for storing new data.

12. A method enacted on a computing system comprising a hardware processor and storage device, the method comprising:

receiving, at a first layer of servers in a cluster in a cloud computing system, a request to store data and associated metadata in the cluster, the first layer of servers configured to authenticate and authorize the request;

compressing the data and the metadata at the first layer of servers upon authentication and authorization of the request;

encrypting the compressed data and metadata at the first layer of servers when encryption is enabled;

sending the request and the encrypted data and metadata to a second layer of servers in the cluster, the second layer of servers configured to store data structures used to manage data storage in a third layer of servers in the cluster;

distributing the request and the encrypted data and metadata to the third layer of servers using the data structures; and

storing the encrypted data and metadata in the third layer of servers, wherein encrypting the data and the metadata at the first layer of servers reduces latency experienced by an end user associated with transferring the data and the metadata between the first, second, and third layers of servers.

13. The method of claim 12 further comprising:

allowing dynamic enabling and disabling of the encryption;

appending the encrypted data and metadata to unencrypted data and metadata stored in the third layer of servers when the encryption is enabled; and

appending unencrypted data and metadata to the encrypted data and metadata stored in the third layer of servers when the encryption is disabled.

14. The method of claim 13 further comprising:

tracking encryption status of data stored in the third layer of servers; and

indicating, based on the tracked status of the data, whether the data is unencrypted, encrypted, or partially encrypted due to the dynamic enabling and disabling of the encryption.

15. The method of claim 12 further comprising:

generating, when encryption is enabled, a unique key to encrypt each block of data, each unique key being based on a unique seed randomly generated for each block of data and based on a first key assigned to an account to which the data belongs; and

encrypting each block of data using a respective key so that consecutive encrypted blocks appear random.

16. The method of claim 15 further comprising:

storing a seed used to encrypt a block in unencrypted form in a header associated with the encrypted block; and

decrypting the encrypted block using the seed stored in the header associated with the encrypted block.

17. The method of claim 15 further comprising:

encrypting the first key using a second key associated with the cluster; and

rotating the second key.

18. The method of claim 12 further comprising:

writing, using the data structure, a record to a log for each received request to write data; and

reproducing, using the log, the request to write data when a server in the second layer of servers crashes.

19. The method of claim 12 further comprising:

storing the encrypted data in data units in the third layer of servers; and

selectively moving content of a first data unit to a second data unit to make the first data unit available for storing new data.

20. A system comprising:

a hardware processor; and

machine readable instructions, stored on a tangible machine readable device, when executed by the hardware processor, configure the hardware processor to:

receive a request to store data in a cluster in a cloud computing system, the request received at a first layer of servers in the cluster, the first layer of servers configured to authenticate and authorize the request;

compress the data at the first layer of servers upon authentication and authorization of the request;

encrypt, when encryption is enabled, each block of the compressed data at the first layer of servers using a unique key generated based on a unique seed randomly generated for each block of data and based on a first key assigned to an account to which the data belongs;

send the request and the encrypted data to a second layer of servers in the cluster, the second layer of servers configured to store data structures used to manage data storage in a third layer of servers in the cluster, and to distribute the request and the encrypted data to the third layer of servers using the data structures; and

store the encrypted data in the third layer of servers, wherein encrypting the data at the first layer of servers reduces latency experienced by an end user associated with transferring the data between the first, second, and third layers of servers.