Efficient cross device redundancy implementation on high performance direct attached non-volatile storage with data reduction
View Patent ↗A method for implementing cross device redundancy schemes with a single commit by receiving, by a write page allocation unit, a request to allocate data grains; responsive to receiving the request, performing, by the write page allocation unit, an analysis of a predetermined data layout map associated with a grain memory to identify a memory segment; allocating, by the write page allocation unit, a number of data grains to the memory segment, while computing redundancy data associated with the number of data grains; storing the number of data grains and the redundancy data to the memory segment of the grain memory; determining, by the write page allocation unit, whether a storage threshold associated with the grain memory has been satisfied; and responsive to the storage threshold associated with the grain memory being satisfied, transmitting data grains and redundancy data stored in the memory segment to one or more storage devices.
1. A method comprising:
receiving, by a write page allocation unit, a request to allocate data grains;
responsive to receiving the request, performing, by the write page allocation unit, an analysis of a predetermined data layout map associated with a grain memory to identify a memory segment;
allocating, by the write page allocation unit, a number of data grains to the memory segment, while computing redundancy data associated with the number of data grains;
storing the number of data grains and the redundancy data to the memory segment of the grain memory;
determining, by the write page allocation unit, whether a storage threshold associated with the grain memory has been satisfied; and
responsive to the storage threshold associated with the grain memory being satisfied, transmitting data grains and redundancy data stored in the memory segment to one or more storage devices.
2. The method of claim 1 , further comprising:
receiving, by a compression unit from a client device, a data stream; and
compressing, by a compression unit, the data stream into a plurality of data grains.
3. The method of claim 1 , wherein the analysis of the predetermined data layout map associated with the grain memory, further comprises:
identifying an active write page associated with the predetermined data layout map;
retrieving a pointer associated with presently stored data grains in the active write page of the predetermined data layout map;
determining unavailable location data associated with the active write page based on the pointer, the unavailable location data reflecting occupied storage space; and
determining available location data associated with the active write page of the predetermined data layout map based on the pointer.
4. The method of claim 1 , wherein storing the number of data grains and the redundancy data to the memory segment of the grain memory includes storing the redundancy data into one or more pre-provisioned memory buffers of the grain memory.
5. The method of claim 4 , wherein allocating the number of data grains to the memory segment, further comprises computing, valid bits per the one or more pre-provisioned memory buffers based on incoming data grains.
6. The method of claim 1 , further comprising:
recording a starting location associated with the predetermined data layout map for a first data grain associated with the number of data grains being allocated to the memory segment; and
updating the predetermined data layout map associated with the grain memory based on the allocating of the number of data grains.
7. The method of claim 1 , wherein transmitting data grains and redundancy data stored in the memory segment further comprises storing in parallel the data grains and redundancy data across a plurality of storage devices.
8. The method of claim 1 , wherein transmitting data grains and redundancy data stored in the memory segment uses a preconfigured template.
9. The method of claim 1 , wherein transmitting data grains and redundancy data stored in the memory segment, further comprises:
generating, a template for transmitting data grains and redundancy data to a plurality of storage devices; and
transferring data grains and redundancy data directly from the grain memory to the plurality of storage devices using the template.
10. The method of claim 9 , wherein the template comprises at least an offset and a number of bytes associated with the data grains and redundancy data.
11. The method of claim 1 , wherein the storage threshold is 32K bytes.
12. The method of claim 1 , wherein the grain size is 512 bytes.
13. The method of claim 1 , wherein allocating the number of data grains to the memory segment is performed simultaneously with computing redundancy data associated with the number of data grains.
14. A system for storage and data reduction comprising:
a compression unit having an input and an output for compressing data, the input of the compression unit coupled to receive a data stream, the compression unit compressing the data stream to produce data grains;
a write page allocation unit having an input and an output for transferring data grains to one or more storage devices and generating redundancy data, the input of the write page allocation unit coupled to the output of the compression unit to receive data grains, the output of write page allocation unit coupled to the one or more storage devices; and
a grain memory for temporarily storing data and having an input and an output, the input coupled to the write page allocation unit and the output coupled to the one or more storage devices.
15. The system of claim 14 further comprising an encryption unit having in an input and an output for encoding the data to produce an encoded data stream, the input of the encryption unit coupled to receive the data stream and the output of the encryption unit coupled to an input of the write page allocation unit.
16. The system of claim 14 wherein the write page allocation unit simultaneously computes redundancy data associated with a number of data grains and allocates the number of data grains to the grain memory.
17. The system of claim 14 wherein the one or more storage devices are a plurality of solid state storage devices.
18. The system of claim 14 wherein the write page allocation unit stores a number of data grains and the redundancy data in parallel across multiple solid state storage devices.
19. The system of claim 14 wherein portions of the grain memory are pre-provisioned for the redundancy data.
20. The system of claim 14 wherein the write page allocation unit uses a single commit to store the data grains and redundancy data to the one or more storage devices.
21. The system of claim 14 further comprising:
allocation logic for analyzing a predetermined data layout map associated with the grain memory to identify a memory segment of the grain memory for temporary storage of data grains, the allocation logic coupled to the compression unit and the grain memory; and
a data queue for temporarily storing grains until they can be stored in the grain memory, the data queue having an input and an output, the input coupled to the data queue coupled to the compression unit and the output of the data queue coupled to the allocation logic.
22. The system of claim 21 wherein the analyzing includes identifying an active write page associated with the predetermined data layout map; retrieving a pointer associated with presently stored data grains in the active write page of the predetermined data layout map; determining unavailable location data associated with the active write page based on the pointer, the unavailable location data reflecting occupied storage space; and determining available location data associated with the active write page of the predetermined data layout map based on the pointer.
23. The system of claim 21 further comprising a DMA command unit for transferring data from the grain memory directly to the one or more storage devices, the DMA command unit coupled for communication with the allocation logic and the one or more storage devices.
24. The system of claim 23 , where in the DMA command unit uses a preconfigured template to transfer data grains and redundancy data from grain memory to the one or more storage devices.