Error correction methods for computational SSD supporting rapid file semantic search
Devices and methods to implement semantic searching on SSD through a computational SSD system that distributes computing to each NAND flash die of the SSD while the SSD controller handles the results aggregation with new on-die computation logic circuits to provide on device file semantic search are disclosed herein. The computational SSD system can read file feature vectors from multiple dies to the SSD controller, and if needed, these feature vectors may be buffered in DRAM and controller handles distance computing. Local, on-die AI/ML processing units may perform, for example, computation and comparison operations and pass the processing scores and results to the SSD controller. The SSD controller aggregates results from all dies and returns the result to the host. The feature vector store size, circuitry and number of on-die AI/ML processing units may be configured as needed to adapt to different tasks, system constraints, and/or feature vector sizes.
1. A device comprising:
a processor;
a memory array comprising a plurality of memory devices; and
a controller communicatively coupled to the memory array;
one or more memory structures configured to store raw file data and feature data corresponding to the raw file data;
wherein the feature data comprises at least one of: dense feature vectors based on error-free data and hyperdimensional vectors;
wherein each of the one or more memory structures utilizes an error correction algorithm for storing and accessing the feature data; and
wherein each of the one or more memory structures comprises a machine learning processing units configured to process the feature data.
2. The device of claim 1 , wherein the dense feature vectors are stored in separate blocks from raw file data.
3. The device of claim 1 , wherein the feature data is distributed within the one or more memory structures.
4. The device of claim 1 , wherein the error correction algorithm is implemented as an on-die Error Correction Code (ECC) with each of the one or more memory structures having ECC.
5. The device of claim 1 , wherein the feature data comprises hyperdimensional vectors, and at least two copies of the hyperdimensional vectors are stored on each of the one or more memory structures, a first copy of the hyperdimensional vectors stored without using an error correction algorithm, and a second copy stored using an error correction algorithm.
6. The device of claim 5 , wherein the first copy of the hyperdimensional vector is provided to the machine learning processing unit, and the second copy of the hyperdimensional vector is provided for correcting errors in the first copy.
7. A method comprising:
storing raw file data in one or more memory structures;
storing feature data corresponding to the raw file data in the corresponding one or more memory structures;
loading feature data from at least one of the one or more memory structures;
wherein the feature data comprises at least one of: dense feature vectors based on error-free data and hyperdimensional vectors;
generating one or more databases from the feature data;
loading the raw file data from at least one of the one or more memory structures; and
utilizing an error correction algorithm for storing and accessing the feature data.
8. The method of claim 7 , wherein storing feature data comprises storing the dense feature vectors in separate blocks from the raw file data.
9. The method of claim 7 , further comprising distributing the feature data within the one or more memory structures.
10. The method of claim 7 , wherein the error correction algorithm is implemented as an on-die Error Correction Code (ECC) with each of the one or more memory structures having ECC.
11. The method of claim 7 , wherein the feature data comprises hyperdimensional vectors, and storing feature data comprises: storing at least two copies of the hyperdimensional vectors on each of the one or more memory structures, a first copy of the hyperdimensional vectors being stored without using an error correction algorithm, and a second copy being stored using an error correction algorithm.
12. The method of claim 7 , wherein each of the one or more memory structures includes a machine learning processing unit, and wherein a first copy of the hyperdimensional vector is provided to the machine learning processing unit, and a second copy of the hyperdimensional vector is provided for correcting errors in the first copy.