Methods and systems for object level de-duplication for solid state devices
In one aspect, the present disclosure relates to a method of de-duplicating data in a solid state storage device. The method can include receiving a block of data to be written to a solid state storage device, wherein the block of data comprises header portion and a payload, wherein the header portion comprises context information; and determining whether the payload should be de-duplicated prior to storage, based on the context information stored within the header portion; if the payload is determined to be de-duplicated, de-duplicating the payload; and storing the de-duplicated payload to the solid state storage device.
1. A method for providing object level de-duplication for solid state storage devices (SSDs), comprising:
receiving, at a solid state storage device (SSD), a block of data comprising a header portion and a payload, wherein the header portion comprises context information indicative that the payload comprises non-random data, and wherein the payload is received from a first location within memory of the SSD;
determining, by the SSD, that the payload should be de-duplicated prior to storage based solely on the context information stored within the header portion without analyzing the payload;
randomizing the non-random data of the payload;
de-duplicating, at the SSD, the payload comprising the randomized data; and
storing the de-duplicated payload to a second location within the memory of the SSD,
wherein the de-duplicating is performed when the SSD is moving the payload from the first location to the second location for garbage collection, and
wherein the header portion is provided by an application layer software program that had previously accessed the payload.
2. The method of claim 1 , wherein the context information is provided from hierarchical layers above the SSD.
3. The method of claim 1 , wherein the payload comprises a plurality of sub-objects.
4. The method of claim 3 , wherein the de-duplicating the payload comprises de-duplicating the payload based on a correlation between sub-objects of other payloads.
5. The method of claim 1 , further comprising, before storing, separating out the header portion from the payload and storing the header portion and the payload in separate locations on the SSD.
6. The method of claim 1 , further comprising deriving the context information from the header portion by analyzing the header portion in view of a known structure of the header portion without having an application layer software program embed additional contextual information into the header portion.
7. The method of claim 1 , wherein the context information is uniquely associated with the payload.
8. A solid state device (SSD) providing object level de-duplication, comprising:
a host interface for communicating to a host device;
one or more flash memories for storing data; and
a de-duplication engine comprising instructions stored in a non-transitory computer readable medium, wherein the instructions, when executed by a processor, are configured to cause:
receiving a block of data, wherein the block of data comprises a header portion and a payload, wherein the header portion comprises context information indicative of whether the payload comprises random data, and wherein the payload is received from a first location within the one or more flash memories;
determining the payload should be de-duplicated prior to storage based solely on the context information stored within the header portion without analyzing the payload;
randomizing data of the payload;
de-duplicating the payload comprising the randomized data; and
storing the de-duplicated payload to a second location within the one or more flash memories,
wherein the de-duplicating is performed when the SSD is moving the payload from the first location to the second location for garbage collection, and
wherein the header portion is provided by an application layer software program that had previously accessed the payload.
9. The solid state device of claim 8 , wherein the instructions are configured to determine that the payload should be de-duplicated based on the context information indicating that the payload is non-random.
10. The solid state device of claim 8 , wherein the context information is provided from hierarchical layers above the SSD.
11. The solid state device of claim 8 , wherein the payload comprises a plurality of sub-objects.
12. The solid state device of claim 11 , wherein the de-duplicating the payload comprises de-duplicating the payload based on a correlation between sub-objects of other payloads.
13. The solid state device of claim 8 , wherein the instructions are further configured to cause, before storing:
separating out the header portion from the payload and storing the header portion and the payload in separate locations on the SSD.
14. A solid state device (SSD) providing object level de-duplication, comprising:
means for communicating to a host device;
means for storing data;
means for receiving a block of data, wherein the block of data comprises a header portion and a payload, wherein the header portion comprises context information indicative of whether the payload comprises random data, and wherein the means for receiving comprises means for receiving the payload from a first location within the means for storing data;
means for determining the payload should be de-duplicated prior to storage based solely on the context information stored within the header portion without analyzing the payload;
means for randomizing data of the payload;
means for de-duplicating the payload comprising the randomized data; and
means for storing the de-duplicated payload to a second location within the means for storing data,
wherein the means for de-duplicating the payload comprises means for de-duplicating the payload when the SSD is moving the payload from the first location to the second location for garbage collection, and
wherein the header portion is provided by an application layer software program that had previously accessed the payload.
15. The solid state device of claim 14 , wherein the context information is provided from hierarchical layers above the SSD.
16. The solid state device of claim 14 , wherein the payload comprises a plurality of sub-objects.
17. The solid state device of claim 16 , wherein the means for de-duplicating the payload comprises means for de-duplicating the payload based on a correlation between sub-objects of other payloads.
18. The solid state device of claim 14 , further comprising means for separating out the header portion from the payload, and wherein the means for storing the de-duplicated payload comprises means for storing the header portion and the payload in separate locations on the means for storing data.
19. The solid state device of claim 14 , wherein the context information indicates that the payload is non-random.