System and methods of managing shared keys in a computer cluster with high availability

1. A method for managing a shared key in a cluster of computers, the method comprising:

transmitting an encrypted node key that has been encrypted using a master key from a first computer in the cluster of computers to a second computer in the cluster of computers so that the encrypted node key is decrypted using the master key at the second computer to obtain a node key of the first computer, wherein the encrypted node key is persistently stored in a storage accessible by the first computer;

receiving the node key from the second computer at the first computer in response to transmission of the encrypted node key; and

decrypting an encrypted shared key using the node key at the first computer to obtain the shared key, wherein the encrypted shared key is persistently stored in the storage accessible by the first computer.

2. The method of claim 1 , further comprising, before transmitting the encrypted node key to the second computer, determining that a key management service is not available for the first computer to obtain the master key from the key management service.

3. The method of claim 1 , further comprising transmitting an identification of the master key with the encrypted node key from the first computer to the second computer so that the correct master key is used to decrypt the encrypted node key at the second computer.

4. The method of claim 1 , further comprising, before decrypting the encrypted shared key at the first computer, calculating a hash of the node key received from the second computer at the first computer and checking the calculated hash of the node key with a stored hash of the node key, wherein the stored hash of the node key is persistently stored in a storage accessible by the first computer.

5. The method of claim 1 , further comprising, before transmitting the encrypted node key to the second computer, rebooting the first host computer that removes the shared key from volatile memory of the first host computer.

6. The method of claim 1 , further comprising:

Receiving an identification of the master key and a master-key encrypted shared key from a cluster management server at the first computer;

obtaining the master key from a key management service using the identification of the master key;

decrypting the master-key encrypted shared key using the master key at the first computer to obtain the shared key;

generating the node key at the first computer;

encrypting the node key using the master key at the first computer;

calculating a hash of the node key at the first computer;

encrypting the shared key using the node key to produce the encrypted shared key; and

persistently storing the encrypted node key, the encrypted shared key and the hash of the node key in the storage.

7. The method of claim 1 , further comprising:

obtaining the master key from a key management service using an identification of the master key;

generating a new node key at the first computer;

encrypting the new node key using the master key at the first computer to produce an encrypted new node key;

calculating a hash of the new node key at the first computer;

encrypting the shared key using the new node key to produce a new encrypted shared key; and

persistently storing the encrypted new node key, the new encrypted shared key and the hash of the new node key in the storage.

8. A non-transitory computer-readable storage medium containing program instructions for managing a shared key in a cluster of computers, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform steps comprising:

transmitting an encrypted node key that has been encrypted using a master key from a first computer in the cluster of computers to a second computer in the cluster of computers so that the encrypted node key is decrypted using the master key at the second computer to obtain a node key of the first computer, wherein the encrypted node key is persistently stored in a storage accessible by the first computer;

receiving the node key from the second computer at the first computer in response to transmission of the encrypted node key; and

decrypting an encrypted shared key using the node key at the first computer to obtain the shared key, wherein the encrypted shared key is persistently stored in the storage accessible by the first computer.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the steps further comprise, before transmitting the encrypted node key to the second computer, determining that a key management service is not available for the first computer to obtain the master key from the key management service.

10. The non-transitory computer-readable storage medium of claim 8 , wherein the steps further comprise transmitting an identification of the master key with the encrypted node key from the first computer to the second computer so that the correct master key is used to decrypt the encrypted node key at the second computer.

11. The non-transitory computer-readable storage medium of claim 8 , wherein the steps further comprise, before decrypting the encrypted shared key at the first computer, calculating a hash of the node key received from the second computer at the first computer and checking the calculated hash of the node key with a stored hash of the node key, wherein the stored hash of the node key is persistently stored in a storage accessible by the first computer.

12. The non-transitory computer-readable storage medium of claim 8 , wherein the steps further comprise, before transmitting the encrypted node key to the second computer, rebooting the first computer that removes the shared key from volatile memory of the first host computer.

13. The non-transitory computer-readable storage medium of claim 8 , wherein the steps further comprise:

receiving an identification of the master key and a master-key encrypted shared key from a cluster management server at the first computer;

obtaining the master key from a key management service using the identification of the master key;

decrypting the master-key encrypted shared key using the master key at the first computer to obtain the shared key;

generating the node key at the first computer;

encrypting the node key using the master key at the first computer;

calculating a hash of the node key at the first computer;

encrypting the shared key using the node key to produce the encrypted shared key; and

persistently storing the encrypted node key, the encrypted shared key and the hash of the node key in the storage.

14. The non-transitory computer-readable storage medium of claim 8 , wherein the steps further comprise:

obtaining the master key from a key management service using an identification of the master key;

generating a new node key at the first computer;

encrypting the new node key using the master key at the first computer to produce an encrypted new node key;

calculating a hash of the new node key at the first computer;

encrypting the shared key using the new node key to produce a new encrypted shared key; and

persistently storing the encrypted new node key, the new encrypted shared key and the hash of the new node key in the storage.

15. A computer in a cluster of computers comprising:

memory; and

a processor configured to:

transmit an encrypted node key that has been encrypted using a master key to a second computer in the cluster of computers so that the encrypted node key is decrypted using the master key at the second computer to obtain a node key of the computer, wherein the encrypted node key is persistently stored in a storage accessible by the computer;

receive the node key from the second computer in response to transmission of the encrypted node key; and

decrypt an encrypted shared key using the node key at the computer to obtain the shared key, wherein the encrypted shared key is persistently stored in the storage accessible by the computer.

16. The computer of claim 15 , wherein the processor is further configured to, before transmitting the encrypted node key to the second computer, determine that a key management service is not available for the first computer to obtain the master key from the key management service.

17. The computer of claim 15 , wherein the processor is further configured to transmit an identification of the master key with the encrypted node key from the first computer to the second computer so that the correct master key is used to decrypt the encrypted node key at the second computer.

18. The computer of claim 15 , wherein the processor is further configured to, before decrypting the encrypted shared key at the first computer, calculate a hash of the node key received from the second computer at the first computer and checking the calculated hash of the node key with a stored hash of the node key, wherein the stored hash of the node key is persistently stored in a storage accessible by the first computer.

19. The computer of claim 15 , wherein the processor is further configured to, before transmitting the encrypted node key to the second computer, reboot the first host computer that removes the shared key from volatile memory of the first host computer.

20. The computer of claim 15 , wherein the processor is further configured to:

receive an identification of the master key and a master-key encrypted shared key from a cluster management server;

obtain the master key from a key management service using the identification of the master key;

decrypt the master-key encrypted shared key using the master key to obtain the shared key;

generate the node key;

encrypt the node key using the master key;

calculate a hash of the node key;

encrypt the shared key using the node key to produce the encrypted shared key; and

persistently store the encrypted node key, the encrypted shared key and the hash of the node key in the storage.