Block device interface using non-volatile pinned memory
A method comprising: receiving, at a block device interface, an instruction to write data, the instruction comprising a memory location of the data; copying the data to pinned memory; performing, by a vector processor, one or more invertible transforms on the data; and writing the data from the pinned memory to one or more storage devices asynchronously; wherein the pinned memory of the data corresponds to a location in pinned memory, the pinned memory being accessible by the vector processor and one or more other processors.
1. A method comprising:
receiving, at a block device interface, an instruction to write data, the instruction comprising a memory location of the data;
copying the data to pinned memory located in main memory;
performing, by a vector processor, one or more invertible transforms on the data;
confirming that the data has been written to one or more storage devices in response to copying the data to the pinned memory located in the main memory before writing the data to one or more storage devices; and
writing the data from the pinned memory located in the main memory to the one or more storage devices asynchronously;
wherein the pinned memory of the data corresponds to a location in pinned memory, the pinned memory being accessible by the vector processor and one or more other processors;
wherein the pinned memory corresponds to non-volatile memory;
wherein the pinned memory is configured as a write-back cache such that writing the data to the one or more storage devices is deferred until space is needed in the write-back cache or until a level of clean capacity retained by the write-back cache falls below a threshold; and
wherein writing the data from the pinned memory to one or more storage devices asynchronously comprises: determining that a part of the write-back cache associated with the data is dirty; writing the data to one or more storage devices; and marking the part of the write-back cache as clean.
2. The method of claim 1 , wherein the confirming occurs before performing one or more invertible transforms on the data.
3. The method of claim 1 , wherein the write-back cache comprises a plurality of partitions, each partition being associated with a source.
4. The method of claim 1 , wherein the non-volatile memory comprises: a plurality of non-volatile memory units; and wherein the data in the pinned memory is in each of the plurality of non-volatile memory units.
5. The method of claim 4 , wherein the plurality of non-volatile memory units comprises:
one or more pairs of NV-DIMMs.
6. The method of claim 4 , further comprising:
determining that a first of the plurality of non-volatile memory units has been replaced with a replacement; and
mirroring the data from another of the plurality of non-volatile memory units to the replacement.
7. The method of claim 1 , wherein the block device interface comprises a block device interface for a meta-device, the meta-device corresponding to the one or more storage devices.
8. The method of claim 1 :
wherein receiving an instruction to write data comprises:
receiving an instruction to modify data stored on the one or more storage devices; and
reading the data stored on the one or more storage devices to the non-volatile write-back cache; and
wherein writing the data to a non-volatile write-back cache comprises:
modifying the data in the non-volatile write-back cache based on the instruction.
9. The method of claim 1 , further comprising:
providing the block device interface.
10. The method of claim 1 , wherein performing, by a vector processor, one or more invertible transforms on the data comprises one or more of:
encryption;
encoding;
compressing;
deduplicating; or
logging.
11. The method of claim 1 , further comprising:
receiving, at a block device interface, an instruction to read data from one or more storage devices;
writing the data from one or more storage devices to a memory location; and
performing, by a vector processor, one or more invertible transforms on the data at the memory location;
wherein the memory location of the data corresponds to a location in pinned memory, the pinned memory being accessible by the vector processor and one or more other processors; and
wherein the pinned memory corresponds to non-volatile memory.
12. The method of claim 11 , wherein the block device interface comprises a block device interface for a meta-device, the meta-device corresponding to the one or more storage devices.
13. The method of claim 11 , further comprising:
providing the block device interface.
14. The method of claim 11 , wherein performing, by a vector processor, one or more invertible transforms on the data comprises one or more of:
decryption;
decoding;
decompressing;
un-deduplicating; or
logging.
15. A system comprising:
one or more vector processors;
one or more CPUs; and
a memory comprising pinned memory;
wherein the memory comprises instructions which, when executed by the one or more vector processors and/or the one or more CPUs, configure the one or more vector processors and/or the one or more CPUs to perform the method of claim 1 .
16. One or more non-transitory computer readable media comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 .