Method and apparatus for content derived data placement in memory
Apparatus and method for placing data based on the content of the data in random access memory such that indexing operations are not required. A strong (e.g., cryptographic) hash is applied to a data element resulting in a signature. A weaker hash function is then applied to the signature to generate a storage location in memory for the data element. The weaker hash function assigns multiple data elements to the same storage location while the signature comprises a unique identifier for locating a particular data element at this location. In one embodiment a plurality of weak hash functions are applied successively to increase storage space utilization. In other embodiments, the assigned storage location can be determined by one or more attributes of the data element and/or the storage technology, e.g, long-lived versus short-lived data and/or different regions of the memory having different performance (e.g., access latency memory lifetime) characteristics.
1. A computer-implemented method of storing data in random access memory based on the content of the data such that indexing operations are not required, the data comprising a plurality of data elements and the random access memory having a plurality of physical storage locations, each physical storage location able to store a plurality of data elements, the method comprising:
generating a signature for a data element by applying a strong hash function to the data element, the signature comprising a unique identifier for finding the data element among a plurality of data elements stored in an assigned physical storage location in the random access memory;
assigning a physical storage location in the random access memory for the data element by applying a first weak hash function to the signature and storing the data element at the assigned physical storage location, wherein the first weak hash function assigns multiple data elements to the same physical storage location;
storing the signature for use in a subsequent read operation; and
retrieving the stored signature for a data element to be read, applying a second weak hash function to the retrieved signature to generate an assigned physical storage location and reading the assigned physical storage location, wherein reading comprises inspecting the data elements stored at the assigned physical storage location for a match with the signature of the data element to be read,
wherein the first weak hash function and the second weak hash function are different or the same.
2. The method of claim 1 , wherein:
the assigning includes determining a physical storage location in the random access memory for the data element by successively applying a plurality of weak hash functions until a physical storage location having free space is located.
3. The method of claim 1 , wherein:
the random access memory includes regions with different performance characteristics each having a plurality of physical storage locations; and
the assigning includes selecting one or more different weak hash functions that target physical storage locations in different regions in random access memory having different performance characteristics to preferentially assign the data element to a select region of physical storage locations in the random access memory.
4. The method of claim 3 , wherein each data element has one or more attributes and the selecting is based on one or more attributes of the data element.
5. The method of claim 3 , wherein:
the performance characteristic is access time.
6. The method of claim 3 , wherein:
the selecting is based on an expected access requirement of the data element.
7. The method of claim 3 , wherein:
the selecting is based on an expected lifetime of the data element in the random access memory.
8. The method of claim 3 , wherein:
the selecting is based on an expected referencing requirement of the data element.
9. The method of claim 1 , wherein the random access memory comprises uniform random access memory having a constant worst case access time.
10. The method of claim 1 , including:
maintaining a partial signature cache comprising a subset of bits of each signature.
11. The method of claim 10 , using the partial signature cache to reduce access time to the data elements stored in the random access memory.
12. The method of claim 1 , wherein each data element is 4, 8, 16, or 256 KB in size.
13. The method of claim 1 , wherein the strong hash function is a cryptographic hash function.
14. The method of claim 1 , comprising utilizing a first in, first out order to store multiple data elements assigned to the same physical storage location.
15. The method of claim 1 , wherein the reading further comprises:
if a matching signature is found at the assigned storage location, returning the data element from the assigned storage location, and if a matching signature is not found at the assigned storage location, applying one or more different weak hash functions to the signature of the data element to be read to generate one or more different assigned storage locations and inspecting the contents of the one or more different assigned storage locations for a match.
16. The method of claim 1 , further comprising writing a data element to the random access memory, wherein the writing includes:
generating the signature and applying the first weak hash function to assign the physical storage location for the data element to be written,
reading the contents of the assigned physical storage location,
if there is an available slot at the assigned physical storage location based on the reading, writing the data element to the assigned physical storage location, and
if there is no available slot at the assigned physical storage location based on the reading, moving the least recently accessed data element at the assigned physical storage location to other storage or deleting it and writing the data element to be written to the assigned physical storage location.
17. The method of claim 1 , wherein:
the assigning includes determining a physical storage location in the random access memory for the data element by successively applying a plurality of weak hash functions until a physical storage location having free space is located, and
each data element is 4, 8, 16, or 256 KB in size.
18. The method of claim 1 , wherein
the random access memory includes regions with different performance characteristics each having a plurality of physical storage locations,
the assigning includes selecting one or more different weak hash functions that target physical storage locations in different regions in the random access memory having different performance characteristics to preferentially assign the data element to a select region of physical storage locations in the random access memory, and
each data element is 4, 8, 16, or 256 KB in size.
19. A computer system storing data in random access memory based on the content of the data such that indexing operations are not required, the data comprising a plurality of data elements and the random access memory having a plurality of physical storage locations, the system comprising:
a server having one or more processors and random access memory storing one or more programs for execution by the one or more processors;
a signature generating program executed by the one or more processors for generating a signature for each of a plurality of data elements by applying a strong hash function to the data element, wherein the signature comprises a unique identifier of the data element for finding the data element among a plurality of data elements stored in an assigned physical storage location;
a location assigning program executed by the one or more processors for applying a first weak hash function to assign a physical storage location in random access memory for the data element, wherein the random access memory has a plurality of physical storage locations and each physical storage location is able to store a plurality of data elements, and wherein a second weak hash function assigns multiple data elements to the same physical storage location;
a storing program executed by the one or more processors for storing the data element at the assigned physical storage location of the random access memory, and storing the signature for use in a subsequent read operation; and
a reading program executed by the one or more processors for retrieving a stored signature for a data element to be read and applying the second weak hash function to the retrieved stored signature to generate an assigned physical storage location to be inspected for the data element to be read, wherein the first weak hash function and the second weak hash function are different or the same.
20. The system of claim 19 , further including:
the random access memory includes regions with different performance characteristics each having a plurality of physical storage locations; and
a selection program executed by the one or more processors for selecting one or more different weak hash functions that target physical storage locations in different regions in the random access memory having different performance characteristics to preferentially assign the data element to a select region of physical storage locations in the random access memory.
21. The system of claim 20 , wherein:
each data element has one or more attributes and the selection is based on one or more attributes of the data element.
22. The system of claim 20 , wherein:
the random access memory comprises uniform random access memory having a constant worst case access time.
23. The system of claim 19 , further including:
the random access memory includes regions with different performance characteristics each having a plurality of physical storage locations;
a selection program executed by the one or more processors for selecting one or more different weak hash functions that target physical storage locations in different regions in the random access memory having different performance characteristics to preferentially assign the data element to a select region of physical storage locations in the random access memory, and
wherein each data element is 4, 8, 16, or 256 KB in size.
24. A non-transitory machine readable medium storing instructions executable by processing resource, the non-transitory machine readable medium comprising:
instructions to generate a signature for a data element of a plurality of data elements by applying a strong hash function to the data element, wherein the signature comprises a unique identifier of the data element for finding the data element among a plurality of data elements stored in an assigned physical storage location;
instructions to apply a first weak hash function to assign a physical storage location in random access memory for the data element, wherein the random access memory has a plurality of physical storage locations and each physical storage location is able to store a plurality of data elements, and wherein the weak hash function assigns multiple data elements to the same physical storage location;
instructions to store the data element at the assigned physical storage location of the random access memory;
instructions to store the signature for use in a subsequent read operation;
instructions to retrieve a stored signature for a data element to be read; and
instructions to apply a second weak hash function to the retrieved stored signature to generate an assigned physical storage location to be inspected for the data element to be read,
wherein the first weak hash function and the second weak hash function are different or the same.
25. The non-transitory machine readable medium of claim 24 , further comprising instructions to select from among different weak hash functions that target physical storage locations in different regions in the random access memory having different performance characteristics to preferentially assign the data element to a select region of physical storage locations in the random access memory.
26. The non-transitory machine readable medium of claim 24 , further comprising instructions to successively apply a plurality of weak hash functions until a physical storage location having free space is located.
27. The non-transitory machine readable medium of claim 24 , further comprising instructions to read the generated assigned physical storage location, including inspection of data elements stored at the generated assigned physical storage location for a match with the retrieved stored signature of the data element to be read.