Distributed storage system and method of reusing symmetric keys for encrypted message transmissions

1. A computer-implemented method for transmitting storage-related messages between host computers in a distributed storage system, the method comprising:

executing a handshake operation of a first-type communication connection between a source data transport daemon of a source host computer in the distributed storage system and a target data transport daemon of a target host computer in the distributed storage system that results in a symmetric key at each of the source and target data transport daemons;

sending the symmetric key of the source data transport daemon from the source data transport daemon to a source data transport manager of the source host computer;

sending the symmetric key of the target data transport daemon from the target data transport daemon to a target data transport manager of the target host computer;

transmitting storage-related message from the source data transport manager to the target data transport manager through a second-type communication connection between the source and target data transport managers, wherein the storage-related message is encrypted and decrypted using the symmetric keys at the source and target data transport managers; and

transmitting additional storage-related messages from the source data transport manager to the target data transport manager through additional second-type communication connections, wherein the additional storage-related messages are again encrypted and decrypted using the same symmetric keys at the source and target data transport managers.

2. The method of claim 1 , wherein the first-type communication connection between the source and target data transport daemons is a Transport Layer Security (TLS) connection.

3. The method of claim 2 , wherein the second-type communication connection between the source and target data transport managers is a Transmission Control Protocol (TCP) connection or a remote direct memory access (RDMA) connection.

4. The method of claim 1 , wherein the source and target data transport daemons are running in a user space of their respective host computers and the source and target data transport managers are running in a kernel space of their respective host computers.

5. The method of claim 1 , further comprising:

adding a replay count in a header of the message at the source data transport manager, wherein the replay count is equal to a replay count of a previous message sent from the source data transport manager to the target data transport manager plus a predefined value.

6. The method of claim 5 , further comprising:

computing a media authentication code (MAC) at the source data transport manager based on at least the replay count of the message using the symmetric key of the source data transport manager and adding the MAC in the header of the message; and

verifying the MAC in the message at the target data transport manager by comparing the MAC with a verification MAC that is computed based on at least the replay count of the message using the symmetric key of the target data transport manager.

7. The method of claim 5 , further comprising:

when the message is received at the target data transport manager, using the replay count of the message and a previous replay count of a previous message received from the source data transport manager to determine whether to process the received message as a valid message, an invalid message or a duplicate message.

8. The method of claim 1 , further comprising:

performing a rekey operation when a predefined period of time has lapsed or a predefined number of messages been transmitted from the source data transport manager to the target data transport manager since the symmetric keys were derived, the rekey operation including:

executing another handshake operation of the first-type communication connection between the source data transport daemon of the source host computer in the distributed storage system and the target data transport daemon of the target host computer in the distributed storage system that results in a new symmetric key at each of the source and target data transport daemons;

sending the new symmetric key of the source data transport daemon from the source data transport daemon to the source data transport manager of the source host computer; and

sending the new symmetric key of the target data transport daemon from the target data transport daemon to the target data transport manager of the target host computer,

wherein the new symmetric keys of the source and target data transport managers are used to encrypt and decrypt subsequent storage-related messages transmitted from the source data transport manager to the target data transport manager through new second-type communication connections between the source and target data transport managers.

9. A non-transitory computer-readable storage medium containing program instructions for transmitting storage-related messages between host computers in a distributed storage system, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform steps comprising:

executing a handshake operation of a first-type communication connection between a source data transport daemon of a source host computer in the distributed storage system and a target data transport daemon of a target host computer in the distributed storage system that results in a symmetric key at each of the source and target data transport daemons;

sending the symmetric key of the source data transport daemon from the source data transport daemon to a source data transport manager of the source host computer;

sending the symmetric key of the target data transport daemon from the target data transport daemon to a target data transport manager of the target host computer;

transmitting storage-related message from the source data transport manager to the target data transport manager through a second-type communication connection between the source and target data transport managers, wherein the storage-related message is encrypted and decrypted using the symmetric keys at the source and target data transport managers; and

transmitting additional storage-related messages from the source data transport manager to the target data transport manager through additional second-type communication connections, wherein the additional storage-related messages are again encrypted and decrypted using the same symmetric keys at the source and target data transport managers.

10. The computer-readable storage medium of claim 9 , wherein the first-type communication connection between the source and target data transport daemons is a Transport Layer Security (TLS) connection.

11. The computer-readable storage medium of claim 10 , wherein the second-type communication connection between the source and target data transport managers is a Transmission Control Protocol (TCP) connection or a remote direct memory access (RDMA) connection.

12. The computer-readable storage medium of claim 9 , wherein the source and target data transport daemons are running in a user space of their respective host computers and the source and target data transport managers are running in a kernel space of their respective host computers.

13. The computer-readable storage medium of claim 9 , wherein the steps further comprise:

adding a replay count in a header of the message at the source data transport manager, wherein the replay count is equal to a replay count of a previous message sent from the source data transport manager to the target data transport manager plus a predefined value.

14. The computer-readable storage medium of claim 13 , wherein the steps further comprise:

computing a media authentication code (MAC) at the source data transport manager based on at least the replay count of the message using the symmetric key of the source data transport manager and adding the MAC in the header of the message; and

verifying the MAC in the message at the target data transport manager by comparing the MAC with a verification MAC that is computed based on at least the replay count of the message using the symmetric key of the target data transport manager.

15. The computer-readable storage medium of claim 13 , wherein the steps further comprise:

when the message is received at the target data transport manager, using the replay count of the message and a previous replay count of a previous message received from the source data transport manager to determine whether to process the received message as a valid message, an invalid message or a duplicate message.

16. The computer-readable storage medium of claim 9 , wherein the steps further comprise:

performing a rekey operation when a predefined period of time has lapsed or a predefined number of messages been transmitted from the source data transport manager to the target data transport manager since the symmetric keys were derived, the rekey operation including:

executing another handshake operation of the first-type communication connection between the source data transport daemon of the source host computer in the distributed storage system and the target data transport daemon of the target host computer in the distributed storage system that results in a new symmetric key at each of the source and target data transport daemons;

sending the new symmetric key of the source data transport daemon from the source data transport daemon to the source data transport manager of the source host computer; and

sending the new symmetric key of the target data transport daemon from the target data transport daemon to the target data transport manager of the target host computer,

wherein the new symmetric keys of the source and target data transport managers are used to encrypt and decrypt subsequent storage-related messages transmitted from the source data transport manager to the target data transport manager through additional second-type communication connections between the source and target data transport managers.

17. A distributed storage system implemented using at least one hardware processors and a memory comprising: a source host computer that includes a source data transport daemon and a source data transport manager; and a target host computer that includes a target data transport daemon and a target data transport manager, wherein the source and target data transport daemons are configured to execute a handshake operation of a first-type communication connection between the source data transport daemon and the target data transport daemon that results in a symmetric key at each of the source and target data transport daemons, wherein the source data transport daemon is configured to send the symmetric key of the source data transport daemon from the source data transport daemon to the source data transport manager of the source host computer, wherein the target data transport daemon is configured to send the symmetric key of the target data transport daemon from the target data transport daemon to the target data transport manager, and wherein the source data transport manager is configured to encrypt storage-related messages transmitted through multiple second-type communications connection from the source data transport manager to the target data transport manager using the symmetric key of the source data transport manager, wherein the target data transport manager is configured to decrypt the storage-related messages transmitted through the multiple second-type communication connections from the source data transport manager to the target data transport manager using the symmetric key of the target data transport manager.

18. The distributed storage system of claim 17 , wherein the first-type communication connection between the source and target data transport daemons is a Transport Layer Security (TLS) connection.

19. The distributed storage system of claim 17 , wherein the second-type communication connection between the source and target data transport managers is a Transmission Control Protocol (TCP) connection or a remote direct memory access (RDMA) connection.

20. The distributed storage system of claim 17 , wherein the source and target data transport daemons are running in a user space of their respective host computers and the source and target data transport managers are running in a kernel space of their respective host computers.