In-memory database-managed container volume replication
In an example embodiment, a solution is used to provide container volume replication via a container storage replication log and volume buffer synchronization, which is built on top of a container cloud platform whose container metadata and replication runtime configuration are all managed by a storage manager (a service orchestrated by its job scheduler and service orchestrator). This container volume replication ensures the data security for a long-running service in the container. In the case of any disaster, the in-memory database and application data inside of the container can be recovered via volume replication. This provides container volume replication for long-running containerized applications whose states keep changing.
1. A system comprising:
at least one hardware processor; and
a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
obtaining a container replication service stored as a container in a container cloud, the container cloud having a container microservice connected to the container, the container comprising a storage replicator and a volume manager;
executing the container replication service to cause copies of an in-memory database to be stored on both a first host computer system and a second host computer system;
receiving a write operation to the in-memory database;
in response to the receiving:
copying data in the write operation to a kernel buffer of the first host computer system;
storing a header and the data in the write operation to a container storage replication log on the first host computer system;
causing the storage replicator to send the write operation from the kernel buffer of the first host computer system to the second host computer system; and
causing the storage replicator to persist the data in the write operation to the copy of the in-memory database on the first host computer system.
2. The system of claim 1 , wherein the causing the storage replicator to persist the data is performed asynchronously.
3. The system of claim 1 , wherein the operations further comprise:
freeing the kernel buffer once the persisting of the data in the write operation has been completed.
4. The system of claim 1 , wherein the operations further comprise:
receiving a notification from the second host computer system that the write operation has been received in a kernel buffer of the second host computer system.
5. The system of claim 4 , wherein the operations further comprise:
in response to a determination that the second host computer system has crashed before persisting the data from the write operation in the copy of the in-memory database stored on the second host computer system, replaying the write operation from the container storage replication log.
6. The system of claim 4 , wherein the operations further comprise:
in response to a determination that the first host computer system has crashed prior to an acknowledgement being received from the second host computer system that the second host computer system has persisted the data from the write operation in the copy of the in-memory database stored on the second host computer system, replaying the write operation from the container storage replication log.
7. The system of claim 1 , wherein the operations further comprise:
freeing the write operation from the kernel buffer of the first host computer system if space in the kernel buffer of the first host computer system is less than a predetermined threshold.
8. A method comprising:
obtaining a container replication service stored as a container in a container cloud, the container cloud having a container microservice connected to the container, the container comprising a storage replicator and a volume manager;
executing, by at least one hardware processor, the container replication service to cause copies of an in-memory database to be stored on both a first host computer system and a second host computer system;
receiving a write operation to the in-memory database;
in response to the receiving:
copying data in the write operation to a kernel buffer of the first host computer system;
storing a header and the data in the write operation to a container storage replication log on the first host computer system;
causing the storage replicator to send the write operation from the kernel buffer of the first host computer system to the second host computer system; and
causing the storage replicator to persist the data in the write operation to the copy of the in-memory database on the first host computer system.
9. The method of claim 8 , wherein the causing the storage replicator to persist the data is performed asynchronously.
10. The method of claim 8 , further comprising:
freeing the kernel buffer once the persisting of the data in the write operation has been completed.
11. The method of claim 8 , further comprising:
receiving a notification from the second host computer system that the write operation has been received in a kernel buffer of the second host computer system.
12. The method of claim 11 ; further comprising:
in response to a determination that the second host computer system has crashed before persisting the data from the write operation in the copy of the in-memory database stored on the second host computer system, replaying the write operation from the container storage replication log.
13. The method of claim 11 , further comprising:
in response to a determination that the first host computer system has crashed prior to an acknowledgement being received from the second host computer system that the second host computer system has persisted the data from the write operation in the copy of the in-memory database stored on the second host computer system, replaying the write operation from the container storage replication log.
14. The method of claim 1 , further comprising:
freeing the write operation from the kernel buffer of the first host computer system if space in the kernel buffer of the first host computer system is less than a predetermined threshold.
15. A non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
obtaining a container replication service stored as a container in a container cloud, the container cloud having a container microservice connected to the container, the container comprising a storage replicator and a volume manager;
executing the container replication service to cause copies of an in-memory database to be stored on both a first host computer system and a second host computer system;
receiving a write operation to the in-memory database;
in response to the receiving:
copying data in the write operation to a kernel buffer of the first host computer system;
storing a header and the data in the write operation to a container storage replication log on the first host computer system;
causing the storage replicator to send the write operation from the kernel buffer of the first host computer system to the second host computer system; and
causing the storage replicator to persist the data in the write operation to the copy of the in-memory database on the first host computer system.
16. The non-transitory machine-readable medium of claim 15 , wherein the causing the storage replicator to persist the data is performed asynchronously.
17. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
freeing the kernel buffer once the persisting of the data in the write operation has been completed.
18. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
receiving a notification from the second host computer system that the write operation has been received in a kernel buffer of the second host computer system.
19. The non-transitory machine-readable medium of claim 18 , wherein the operations further comprise:
in response to a determination that the second host computer system has crashed before persisting the data from the write operation in the copy of the in-memory database stored on the second host computer system, replaying the write operation from the container storage replication log.
20. The non-transitory machine-readable medium of claim 18 , wherein the operations further comprise:
in response to a determination that the first host computer system has crashed prior to an acknowledgement being received from the second host computer system that the second host computer system has persisted the data from the write operation in the copy of the in-memory database stored on the second host computer system, replaying the write operation from the container storage replication log.