Methods and systems for object level de-duplication for data storage system
In one aspect, the present disclosure relates to a method of de-duplicating data in a solid state storage device. The method can include receiving a block of data to be written to a solid state storage device, wherein the block of data comprises header portion and a payload, wherein the header portion comprises context information; and determining whether the payload should be de-duplicated prior to storage, based on the context information stored within the header portion; if the payload is determined to be de-duplicated, de-duplicating the payload; and storing the de-duplicated payload to the solid state storage device.
1. A data storage system, comprising:
memories; and
one or more controllers configured to cause:
receiving a first data block and receiving a second data block, wherein the first data block comprises first header portions and first payload portions, and the second data block comprises second header portions and second payload portions,
determining whether the first payload portions and the second payload portions are identical by comparing each of the first header portions to a respective one of the second header portions without comparing the first payload portions to the second payload portions,
when one or more of the second header portions are different from one or more of the first header portions, storing, in the memories, one or more of the second payload portions associated with the one or more of the second header portions, preventing from storing, in the memories, other one or more portions of the second payload portions, and storing, in the memories, one or more references to other one or more portions of the first payload portions,
wherein:
a first context information of a respective one of the first header portions is uniquely associated with a respective one of the first payload portions; and
a second context information of a respective one of the second header portions is uniquely associated with a respective one of the second payload portions.
2. The data storage system of claim 1 , wherein the other one or more portions of the second payload portions are identical to the other one or more portions of the first payload portions.
3. The data storage system of claim 1 , wherein the comparing comprises comparing each of the first header portions to a respective one of the second header portions without hashing the first payload portions or the second payload portions.
4. The data storage system of claim 1 , wherein the first and second header portions are provided by an application layer software program that had previously accessed the first and second payload portions.
5. The data storage system of claim 1 , wherein the second header portions are provided by an application layer software program that had previously accessed the second payload portions.
6. The data storage system of claim 1 , wherein:
the first data block comprises third header portions and third payload portions;
the second data block comprises fourth header portions and fourth payload portions;
the third payload portions comprise random data;
the fourth payload portions comprise random data; and
the one or more controllers are configured to cause determining whether the third payload portions and the fourth payload portions are identical by comparing a subset of the random data of the third payload portions to a subset of the random data of the fourth payload portions without comparing remaining random data of the third payload portions to remaining random data of the fourth payload portions and without hashing the third payload portions or the fourth payload portions.
7. The data storage system of claim 1 , wherein the one or more controllers are configured to cause storing the one or more of the second payload portions and storing the one or more references during a garbage collection operation.
8. The data storage system of claim 1 , wherein:
each of the first header portions comprises the first context information associated with the respective one of the first payload portions;
each of the second header portions comprises the second context information associated with the respective one of the second payload portions;
the one or more controllers are configured to cause: deriving the first context information from the first header portions by analyzing the first header portions in view of a predetermined structure of the first header portions without having an application layer software program embed additional contextual information into the first header portions.
9. The data storage system of claim 1 , wherein:
the first context information is provided from hierarchical layers above the data storage device.
10. The data storage system of claim 1 , wherein the first data block is previously stored in the memories.
11. An apparatus, comprising:
means for receiving a first data block and receiving a second data block, wherein the first data block comprises first header portions and first payload portions, and the second data block comprises second header portions and second payload portions,
means for determining whether the first payload portions and the second payload portions are identical by comparing each of the first header portions to a respective one of the second header portions without comparing the first payload portions and the second payload portions,
when one or more of the second header portions are different from one or more of the first header portions, means for storing, in memories, one or more of the second payload portions associated with the one or more of the second header portions, preventing from storing, in the memories, other one or more portions of the second payload portions, and storing, in the memories, one or more references to other one or more portions of the first payload portions,
wherein:
a first context information of a respective one of the first header portions is uniquely associated with a respective one of the first payload portions; and
a second context information of a respective one of the second header portions is uniquely associated with a respective one of the second payload portions.
12. The apparatus of claim 11 , wherein the other one or more portions of the second payload portions are identical to the other one or more portions of the first payload portions.
13. The apparatus of claim 11 , wherein the first and second header portions are provided by an application layer software program that had previously accessed the first and second payload portions.
14. The apparatus of claim 11 , wherein the comparing comprises comparing each of the first header portions to a respective one of the second header portions without hashing the first payload portions or the second payload portions.
15. A method for a data storage system, comprising:
receiving a first data block and receiving a second data block, wherein the first data block comprises first header portions and first payload portions, and the second data block comprises second header portions and second payload portions,
determining whether the first payload portions and the second payload portions are identical by comparing each of the first header portions to a respective one of the second header portions without comparing the first payload portions to the second payload portions,
when one or more of the second header portions are different from one or more of the first header portions, storing, in memories, one or more of the second payload portions associated with the one or more of the second header portions, preventing from storing, in the memories, other one or more portions of the second payload portions, and storing, in the memories, one or more references to other one or more portions of the first payload portions,
wherein:
a first context information of a respective one of the first header portions is uniquely associated with a respective one of the first payload portions; and,
a second context information of a respective one of the second header portions is uniquely associated with a respective one of the second payload portions.
16. The method of claim 15 , wherein the other one or more portions of the second payload portions are identical to the other one or more portions of the first payload portions.
17. The method of claim 15 , wherein the comparing comprises comparing each of the first header portions to a respective one of the second header portions without hashing the first payload portions or the second payload portions.
18. The method of claim 15 , wherein the first and second header portions are provided by an application layer software program that had previously accessed the first and second payload portions.
19. The method of claim 15 , wherein:
the first data block comprises third header portions and third payload portions;
the second data block comprises fourth header portions and fourth payload portions;
the third payload portions comprise random data;
the fourth payload portions comprise random data; and
the method comprises: determining whether the third payload portions and the fourth payload portions are identical by comparing a subset of the random data of the third payload portions to a subset of the random data of the fourth payload portions without comparing remaining random data of the third payload portions to remaining random data of the fourth payload portions and without hashing the third payload portions or the fourth payload portions.
20. The method of claim 15 , wherein:
each of the first header portions comprises the first context information associated with the respective one of the first payload portions;
each of the second header portions comprises the second context information associated with the respective one of the second payload portions;
the method comprises: deriving the first context information from the first header portions by analyzing the first header portions in view of a predetermined structure of the first header portions without having an application layer software program embed additional contextual information into the first header portions.