Advanced storage operations for wan optimization
Some embodiments of the invention provide a WAN optimization method for optimizing traffic flows through a WAN connecting multiple sites each having at least one router. The method is performed in a kernel space of a first router at a first site. From a second router at a second site, the method receives, in an optimized data stream originating from a source device at the second site and destined to a destination device at the first site, a file including compressed segments not yet stored in a database of the first router. The method copies the compressed segments to a shared memory used with a decompression program operating in the first router's user space. For each compressed segment, the method receives, from the decompression program, a decompressed segment obtained by decompressing the compressed segment, and stores the decompressed segment in a kernel space memory for use in processing the file.
1. A WAN (wide area network) optimization method for optimizing traffic flows through a WAN that connects a plurality of sites, each of which has at least one router, the method comprising:
in a kernel space of a first router located at a first site:
from a second router located at a second site, receiving a file in an optimized first data stream originating from a source device at the second site and destined to a destination device at the first site, the file comprising a set of one or more compressed segments that are not yet stored in a database of the first router;
copying the set of one or more compressed segments to a shared memory that is used with a decompression program operating in a user space of the first router;
for each particular compressed segment in the set of one or more compressed segments,
receiving, from the user space decompression program, a particular decompressed segment that the user space decompression program obtains by decompressing the particular compressed segment; and
storing the particular decompressed segment in a kernel space memory for use in processing the file and subsequent files received by the first router.
2. The method of claim 1 , wherein the decompression program retrieves the set of compressed segments from the shared memory in order to (i) decompress the set of compressed segments, (ii) provide the set of decompressed segments to the kernel space of the first router, and (iii) store the set of decompressed segments in a database in the user space of the first router, wherein the database stores a plurality of decompressed segments for the first router.
3. The method of claim 2 , wherein a database agent of the database writes each decompressed segment in the set of decompressed segments to a disk storage of the first router.
4. The method of claim 3 further comprising:
for each decompressed segment in the set of decompressed segments,
receiving a notification from the user space of the first router that the decompressed segment has been written to the disk storage of the first router; and
based on the notification, deleting the decompressed segment from the kernel space memory of the first router.
5. The method of claim 4 , wherein after deleting the particular decompressed segment from the kernel space memory, the kernel space of the first router processes the file and subsequent files received by the first router by performing an operation to DMA (direct memory access) the particular decompressed segment from the disk storage to the kernel space memory.
6. The method of claim 5 , wherein performing the operation comprises directing an NVMe (non-volatile memory express) device operating in the kernel space of the first router to perform the operation.
7. The method of claim 4 , wherein storing each particular decompressed segment in the kernel space memory further comprises updating a cache of the first router with an entry for the particular decompressed segment, the entry comprising (i) an identifier of the particular decompressed segment, (ii) an address corresponding to a location of the particular decompressed segment in the kernel space memory, and (iii) an indicator value that indicates the particular decompressed segment is stored in the kernel space memory.
8. The method of claim 7 , wherein the notification that the decompressed segment has been written to the disk storage comprises a block address corresponding to a location of the particular decompressed segment in the disk storage, wherein the method further comprises, based on the notification, updating the entry in the cache for the particular decompressed segment to replace the address corresponding to the location of the particular decompressed segment in the kernel space memory with the block address from the notification.
9. The method of claim 8 , wherein deleting the decompressed segment from the kernel space memory of the first router further comprises updating the entry in the cache to replace the indicator value that indicates the particular decompressed segment is stored in the kernel space memory with an indicator value that indicates the particular decompressed segment is not stored in the kernel space memory.
10. The method of claim 1 , wherein processing the file comprises using the set of decompressed segments to reconstruct the file by replacing the set of compressed segment in the file with the set of decompressed segments.
11. The method of claim 10 further comprising sending the reconstructed file to the destination device.
12. The method of claim 1 , wherein receiving, from the user space decompression program, the particular decompressed segment comprises retrieving the particular decompressed segment from the shared memory after the user space decompression program copies the particular decompressed segment to the shared memory for retrieval by the kernel space of the first router.
13. The method of claim 1 , wherein the first router comprises a software router executing on a host computer and the disk storage is a disk storage of the host computer.
14. The method of claim 13 , wherein at least one source or one destination of WAN traffic flows execute on the host computer with the software router.
15. The method of claim 1 , wherein the first router comprises a standalone appliance and the disk storage is a disk storage of the standalone appliance.
16. A non-transitory machine readable medium storing a WAN (wide area network) optimization program for execution by a set of processing units, the WAN optimization program for optimizing traffic flows through a WAN that connects a plurality of sites, each of which has at least one router, the WAN optimization program comprising sets of instructions for:
in a kernel space of a first router located at a first site:
from a second router located at a second site, receiving a file in an optimized first data stream originating from a source device at the second site and destined to a destination device at the first site, the file comprising a set of one or more compressed segments that are not yet stored in a database of the first router;
copying the set of one or more compressed segments to a shared memory that is used with a decompression program operating in a user space of the first router;
for each particular compressed segment in the set of one or more compressed segments,
receiving, from the user space decompression program, a particular decompressed segment that the user space decompression program obtains by decompressing the particular compressed segment; and
storing the particular decompressed segment in a kernel space memory for use in processing the file and subsequent files received by the first router.
17. The non-transitory machine readable medium of claim 16 , wherein the decompression program retrieves the set of compressed segments from the shared memory in order to (i) decompress the set of compressed segments, (ii) provide the set of decompressed segments to the kernel space of the first router, and (iii) store the set of decompressed segments in a database in the user space of the first router, wherein the database stores a plurality of decompressed segments for the first router.
18. The non-transitory machine readable medium of claim 17 , wherein a database agent of the database writes each decompressed segment in the set of decompressed segments to a disk storage of the first router.
19. The non-transitory machine readable medium of claim 18 , the WAN optimization program further comprising sets of instructions for:
for each decompressed segment in the set of decompressed segments,
receiving a notification from the user space of the first router that the decompressed segment has been written to the disk storage of the first router; and
based on the notification, deleting the decompressed segment from the kernel space memory of the first router.
20. The non-transitory machine readable medium of claim 19 , wherein after deleting the particular decompressed segment from the kernel space memory, the WAN optimization program further comprises a set of instructions for processing, at the kernel space of the first router, the file and subsequent files received by the first router by directing an NVMe (non-volatile memory express) device operating in the kernel space of the first router to perform an operation to DMA (direct memory access) the particular decompressed segment from the disk storage to the kernel space memory.