Non-volatile cross-bar memory with on-chip processing circuitry
Exemplary methods and apparatus are disclosed that implement super-sparse image/video compression by storing image dictionary elements within a cross-bar resistive random access memory (ReRAM) array (or other suitable cross-bar NVM array). In illustrative examples, each column of the cross-bar ReRAM array stores the values for one dictionary element (such as one 4×4 dictionary element). Methods and apparatus are described for training (configuring) the cross-bar ReRAM array to generate and store the dictionary elements by sequentially applying patches from training images to the array using an unstructured Hebbian training procedure. Additionally, methods and apparatus are described for compressing an input image by applying patches from the input image to the ReRAM array to read out cross-bar column indices identifying the columns storing the various dictionary elements that best fit the image. This may be done in parallel using a set of ReRAM arrays.
1. A non-volatile memory (NVM) die, comprising:
a cross-bar NVM array formed within the NVM die;
a training image input circuit formed within the NVM die and configured to input a set of training images;
a cross-bar training circuit formed within the NVM die and configured to generate a dictionary of image elements and store the dictionary of image elements within the cross-bar NVM array for use in image compression, wherein the cross-bar training circuit is configured to generate the dictionary of image elements from the set of training images; and
a cross-bar image compression circuit formed within the NVM die and configured to:
apply a patch of an input image to the cross-bar NVM array to identify an image element in the dictionary of image elements that most closely corresponds to the patch;
generate a grayscale value based on the patch; and
output a compressed version of the input image that replaces the patch with the grayscale value and with an indicator of the image element in the dictionary of image elements that most closely corresponds to the patch.
2. The NVM die of claim 1 , wherein the cross-bar training circuit comprises:
a cross-point resistance programming circuit configured to set cross-point resistance values of the cross-bar NVM array to store individual elements of the dictionary of image elements within individual columns of the cross-bar NVM array.
3. The NVM die of claim 2 , wherein the cross-point resistance programming circuit comprises:
an image patch selection circuit configured to select a set of image patches from within each training image of the set of training images; and
a sequential training circuit configured to sequentially apply each image patch of the set of image patches from each training image of the set of training images to the cross-bar NVM array to iteratively configure the cross-point resistance values to generate and store the dictionary of image elements.
4. The NVM die of claim 3 , wherein:
each image patch of the set of image patches comprises an m×m array of M pixels;
the dictionary of image elements comprises N image elements, each image element comprising an m×m array of M pixels;
the cross-bar NVM array is an N×M array comprising N columns, each of which includes M cross-point resistance values; and
the cross-point resistance programming circuit is further configured to set each of the N×M cross-point resistance values of the cross-bar NVM array based on the training images.
5. The NVM die of claim 1 , wherein:
the cross-bar NVM array includes a plurality of individual cross-bar NVM arrays, each configured to store the dictionary of image elements; and
the cross-bar training circuit is further configured to set cross-point resistance values of each individual cross-bar NVM array of the plurality of individual cross-bar NVM arrays based on the training images to store the dictionary of image elements.
6. The NVM die of claim 1 , wherein the cross-bar training circuit comprises:
an unsupervised cross-bar training circuit configured to set cross-point resistance values of the cross-bar NVM array using an unsupervised training procedure.
7. A method for use with a cross-bar non-volatile memory (NVM) array of an NVM die that includes a cross-bar training circuit and a cross-bar image compression circuit, the method comprising:
applying a set of image compression training images to the cross-bar NVM array using the cross-bar training circuit, wherein each of the cross-bar NVM array and the cross-bar training circuit are formed within the NVM die;
determining and setting cross-point values of the cross-bar NVM array based on the image compression training images using the cross-bar training circuit to generate and store a dictionary of image elements within the cross-bar NVM array for use in image compression;
applying a patch of an input image to the cross-bar NVM array using the cross-bar image compression circuit to identify an image element in the dictionary of image elements that most closely corresponds to the patch;
generating a grayscale value based on the patch using the cross-bar image compression circuit; and
outputting a compressed version of the input image from the cross-bar NVM array that replaces the patch with the grayscale value and with an indicator of the image element in the dictionary of image elements that most closely corresponds to the patch.
8. The method of claim 7 , wherein the cross-point values are resistance values set based on the training images to store individual elements of the dictionary of image elements within individual columns of the cross-bar NVM array.
9. The method of claim 8 , wherein applying the set of training images to the cross-bar NVM array comprises sequentially applying each image patch of a set of image patches of each training image of the set of training images to the cross-bar NVM array to iteratively configure the cross-point resistance values.
10. The method of claim 8 , wherein:
the cross-bar NVM array includes a plurality of individual cross-bar NVM arrays, each configured to store the dictionary of image elements; and
the cross-point resistance values of each individual cross-bar NVM array are trained based on the training images to store the dictionary of image elements.
11. The method of claim 8 , wherein determining and setting the cross-point resistance values of the cross-bar NVM array comprises:
(a) selecting a patch of a training image from the set of training images;
(b) modifying resistance values of cross-point elements of the cross-bar NVM array based on the selected patch to conform the resistance values of one column of the cross-bar NVM array to the patch of the image;
(c) repeating (a) and (b) with additional patches of the training image to further modify the resistance values of the cross-point elements to conform the resistance values of the same or other columns of the cross-bar NVM array to the additional patches of the image; and
(d) repeating (a), (b), and (c) with additional training images of the set to further modify the resistance values of the cross-point elements until the resistance values of each particular column of the cross-bar NVM array correspond to one of the image elements of the dictionary of image elements.
12. The method of claim 8 , wherein the cross-point resistance values of the cross-bar NVM array are configured based on the training images using an unsupervised training procedure.
13. The NVM die of claim 1 , wherein:
the patch is one of a plurality of patches that collectively comprise the entire input image; and
the cross-bar image compression circuit is further configured to process each patch of the plurality of patches to generate a compressed version of the entire input image.
14. The NVM die of claim 1 , wherein the indicator comprises a column index within the cross-bar NVM array that identifies the image element in the dictionary of image elements.
15. A data storage device, comprising:
a non-volatile memory (NVM) die comprising a cross-bar NVM array; and
one or more processors configured, individually or in combination, to:
input a set of training images;
generate a dictionary of image elements from the set of training images;
store the dictionary of image elements within the cross-bar NVM array for use in image compression;
apply a patch of an input image to the cross-bar NVM array to identify an image element in the dictionary of image elements that most closely corresponds to the patch;
generate a grayscale value based on the patch; and
output a compressed version of the input image that replaces the patch with the grayscale value and with an indicator of the image element in the dictionary of image elements that most closely corresponds to the patch.
16. The data storage device of claim 15 , wherein:
the patch is one of a plurality of patches that collectively comprise the entire input image; and
the one or more processors are further configured, individually or in combination, to process each patch of the plurality of patches to generate a compressed version of the entire input image.
17. The data storage device of claim 15 , wherein the indicator comprises a column index within the cross-bar NVM array that identifies the image element in the dictionary of image elements.
18. A method for use by a data storage device with a non-volatile memory (NVM) die comprising a cross-bar NVM array, the method comprising:
inputting a set of training images;
generating a dictionary of image elements from the set of training images;
storing the dictionary of image elements within the cross-bar NVM array for use in image compression;
applying a patch of an input image to the cross-bar NVM array to identify an image element in the dictionary of image elements that most closely corresponds to the patch;
generating a grayscale value based on the patch; and
outputting a compressed version of the input image that replaces the patch with the grayscale value and with an indicator of the image element in the dictionary of image elements that most closely corresponds to the patch.
19. The method of claim 18 , wherein:
the patch is one of a plurality of patches that collectively comprise the entire input image; and
the data storage device processes each patch of the plurality of patches to generate a compressed version of the entire input image.
20. The method of claim 18 , wherein the indicator comprises a column index within the cross-bar NVM array that identifies the image element in the dictionary of image elements.