Nonvolatile memory systems configured to use deduplication and methods of controlling the same
Systems and methods of determining a similarity between data units in a nonvolatile memory are disclosed. One method includes obtaining first and second data units and dividing the first and second data units into a first plurality of non-overlapping chunks of data and a second plurality of non-overlapping chunks of data. The method further includes determining a first plurality of values and a second plurality of values associated with the chunks, and determining a similarity between the first second data units based on the first plurality values and of the second plurality of values. In one example embodiment, a similarity between an incoming data unit and another data unit is determined based on the number of buckets storing an incoming index value and another index value associated with the another data unit. A plurality of buckets in a table is determined based on a selected plurality of hash values.
1. A method of determining a similarity between data units in a nonvolatile memory system, the method comprising:
receiving an incoming data unit;
assigning an incoming index value to the incoming data unit;
dividing the incoming data unit into a plurality of chunks, the incoming data unit represented by a vector of the plurality of chunks;
determining an incoming plurality of hash values of the plurality of chunks based on a hash function;
determining a selected plurality of hash values from the incoming plurality of hash values;
determining a plurality of buckets in a table based on the selected plurality of hash values, each of the plurality of buckets associated with a different one of the selected plurality of hash values;
storing the incoming index value in at least one of the plurality of buckets;
determining a number of buckets in the plurality of buckets storing the incoming index value and at least another index value associated with at least another data unit; and
determining a similarity between the incoming data unit and the at least another data unit based on the determined number.
2. The method of claim 1 , wherein the incoming data unit is one of a sector and a page.
3. The method of claim 2 , wherein a number of the plurality of chunks is the number of cells in a sector divided by an average size of the plurality of chunks.
4. The method of claim 1 , wherein the selected plurality of hash values includes all of the incoming plurality of hash values.
5. The method of claim 1 , wherein the determining of the selected plurality of hash values includes,
determining a first possible value for the hash function;
determining a threshold based on a product of the first possible value and a constant factor; and
determining the selected plurality of hash values based on the incoming plurality of hash values that exceed the threshold.
6. The method of claim 1 , wherein the determining of the similarity determines the similarity based on whether the determined number exceeds a threshold number.
7. The method of claim 6 , wherein the threshold number is one.
8. The method of claim 1 , further comprising:
writing the incoming data unit into a physical page of a nonvolatile memory based on the similarity, the writing including,
writing a reference page having different chunks of the plurality of chunks, and
writing a difference between the incoming data unit and the at least another data unit.
9. The method of claim 1 , wherein the nonvolatile memory is a flash device.
10. The method of claim 9 , wherein the flash device is a NAND flash device.
11. The method of claim 1 , wherein the determining of the similarity determines the similarity independent of positions of the plurality of chunks in the incoming data unit.
12. The method of claim 1 , wherein the plurality of chunks are non-overlapping.
13. A memory system comprising:
a nonvolatile memory device; and
a memory controller configured to,
receive an incoming data unit,
assign an incoming index value to the incoming data unit,
divide the incoming data unit into a plurality of chunks, the incoming data unit represented by a vector of the plurality of chunks,
determine an incoming plurality of hash values of the plurality of chunks based on a hash function,
determine a selected plurality of hash values from the incoming plurality of hash values,
determine a plurality of buckets in a table based on the selected plurality of hash values, each of the plurality of buckets associated with a different one of the selected plurality of hash values,
store the incoming index value in at least one of the plurality of buckets,
determine a number of buckets in the plurality of buckets storing the incoming index value and at least another index value associated with at least another data unit, and
determine a similarity between the incoming data unit and the at least another data unit based on the determined number.
14. The memory system of claim 13 , wherein the incoming data unit is one of a sector and a page, and
the number of the plurality of chunks is the number of cells in a sector divided by an average size of the plurality of chunks.