Concept for approximate deduplication in storage and memory
Examples relate to an approximative memory deduplication method, a controller apparatus or controller device for a memory or storage controller, a memory or storage controller, a computer system and to a computer program. The approximative memory deduplication method comprises determining a hash value of a data block. The hash value is based on a user-defined approximative hashing function. The approximative memory deduplication method comprises storing a quantized version of the data block based on the hash value using a memory or storage device of the computer system.
1. An apparatus for a memory or storage controller of a computer system, wherein the apparatus comprises:
interface circuitry for communicating with a memory or storage device of the computer system, and
processing circuitry configured to:
determine a hash value based on a user-defined approximative hashing function, wherein the user-defined approximative hashing function:
quantizes content of a data block creating a quantized data block,
hashes the quantized data block to produce the hash value, and
based on the hash value, store the quantized data block using the memory or storage device.
2. The apparatus according to claim 1 wherein the user-defined approximative hashing function defines a user-desired precision for the quantized data block.
3. The apparatus according to claim 1 , wherein the user-defined approximative hashing function is specific to a content type of the data block.
4. The apparatus according to claim 1 wherein the user-defined approximative hashing function is suitable for assigning the data block to a bucket of a plurality of possible buckets.
5. The apparatus according to claim 1 , wherein the quantized data block has a lower precision or lower resolution than the data block.
6. The apparatus according to claim 1 wherein the processing circuitry is configured to receive the data block within a write command via the interface circuitry, wherein the quantized data block is stored upon reception of the data block, wherein the data block is discarded after storing of the quantized data block.
7. The apparatus according to claim 1 , wherein the memory or storage device is a volatile memory device.
8. The apparatus according to claim 1 , wherein the memory or storage device is a persistent storage device.
9. The apparatus according to claim 1 , wherein the data block comprises numerical data and/or sensor data.
10. The apparatus according to claim 1 , wherein the data block comprises non-numerical content, wherein the user-defined approximative hashing function is suitable for assigning the data block to a bucket of a plurality of possible buckets based on the non-numerical content.
11. The apparatus according to claim 1 , wherein the processing circuitry is configured to provide a memory address for accessing the stored quantized data block.
12. The apparatus according to claim 11 wherein the processing circuitry is configured to provide transparent memory access to the stored quantized data block based on the memory address for accessing the stored quantized data block.
13. The apparatus according to claim 11 wherein the processing circuitry is configured to receive a read command comprising the memory address, wherein the processing circuitry is configured to provide the quantized data block within a response to the read command.
14. The apparatus according to claim 1 , wherein the processing circuitry is configured to determine, whether the quantized data block is previously stored within the memory or storage device based on the hash value.
15. The apparatus according to claim 14 , wherein the storing of the quantized data block avoids writing a new data record within the memory or storage device if the quantized data block is previously stored within a data record within the memory or storage device.
16. The apparatus according to claim 15 , wherein the quantized data block is stored within a data record with a reference count for the quantized data block, wherein the storing oft the quantized data block comprises updating the reference count of the data record associated with the quantized data block if the quantized data block is previously stored within the data record within the memory or storage device.
17. A memory or storage controller for a computer system, the memory or storage controller comprising the apparatus according to claim 1 .
18. A non-transitory, computer-readable storage medium including program code, when executed, to cause a machine to perform a method, the method comprising:
determining a hash value based on a user-defined approximative hashing function, wherein the user-defined approximative hashing function:
quantizes content of a data block creating a quantized data block,
hashes the quantized data block to produce the hash value, and
based on the hash value, storing the quantized data block using a memory or storage device of the machine.