Storage device for storing model checkpoints of recommendation deep-learning models
The present disclosure generally relates to utilizing improved DL training models stored in non-volatile memory to optimize data transfer and storage. The proposed system would identify workloads of DNN training and occasionally check the difference rate between successive data transfers (representing successive training iterations of the model). Comparing the difference rate to given thresholds could indicate “recommendation-system” typical use case. In such a case the NAND operating system would apply systematic compression of the data by saving only the changed parameters between successive iteration cycles (“batches”). The host may indicate the checkpoint storage configuration of the training model (every iteration, every several iterations etc. . . . ) and other elements. The system may be efficiently utilized combining the NAND based DNN training interface, adding the checkpoint configuration information to the dedicated interface. The current iteration's model is stored in the NAND, so adding the checkpoint related information is most efficient.
1. A data storage device, comprising:
a memory device; and
a controller coupled to the memory device, wherein the controller is configured to be coupled to a host device, and wherein the controller is further configured to:
receive a first command;
generate logical block address (LBA) to physical block address (PBA) (L2P) mappings for the first command, wherein the L2P mapping is generated based on a result of deep learning (DL) training model using a neural network (NN) structure;
store data of the first command in the memory device;
receive a second command;
determine a difference between the data of the first command and data of the second command;
generate LBA to PBA L2P mappings for the difference, wherein the L2P mapping is generated based on a result of DL training module using the NN structure; and
compress the data of the first command using a compression engine to store the difference in the memory device.
2. The data storage device of claim 1 , wherein the controller is further configured to identify repeating sequential write chunks.
3. The data storage device of claim 2 , wherein the controller is further configured to compare two successive write chunks of the repeating sequential write chunks.
4. The data storage device of claim 3 , wherein the controller is further configured to determine whether the difference between the compared two successive write chunks is less than a threshold.
5. The data storage device of claim 4 , wherein the controller is configured to operate difference based compression upon determining the difference is below the threshold.
6. The data storage device of claim 4 , wherein the controller is configured to operate with non-compressed full-representation storing upon determining the difference is above the threshold.
7. The data storage device of claim 1 , wherein the controller is configured to receive an indication from the host device that compression configuration is applicable.
8. The data storage device of claim 1 , wherein the controller is further configured to compress the data by computing a delta for each iteration of the DL training model and storing the computed delta in the memory device.
9. The data storage device of claim 8 , wherein the delta is relative to an initial DL training model stored in the memory device.
10. The data storage device of claim 1 , wherein the controller is further configured to store deltas for each iteration of the DL training model in the memory device and store an original iteration of the DL training model in NAND or RAM.
11. The data storage device of claim 1 , wherein the memory device is a non-volatile memory device.
12. A data storage device, comprising:
a memory device; and
a controller coupled to the memory device, wherein the controller comprises:
a neural network (NN) command interpretation unit;
a logical block address (LBA) to physical block address (PBA) (L2P) mapping generator coupled to the NN command interpretation unit, wherein the controller is configured to fetch training data and NN parameters from the memory device; and
a compression engine coupled to both the NN command interpretation unit and the L2P mapping generator, wherein the compression engine is configured to compress the fetched training data and NN parameters by storing a delta based on the fetched training data and NN parameters.
13. The data storage device of claim 12 , wherein the NN command interpretation unit is configured to interface with a NN interface command generator disposed in a host device.
14. The data storage device of claim 12 , wherein the NN parameters are key value pair data.
15. The data storage device of claim 12 , wherein the NN parameters are used in a deep learning (DL) training model.
16. The data storage device of claim 15 , wherein one or more parts of the DL training model are disabled.
17. The data storage device of claim 12 , wherein the controller is configured to update one or more weights associated with a deep learning (DL) training model, and wherein the updating is to a same address as a previous read of the one or more weights.
18. A data storage device, comprising:
non-volatile memory means; and
a controller coupled to the non-volatile memory means, wherein the controller is configured to:
store neural network (NN) parameters in the non-volatile memory means;
performing a deep learning (DL) training model;
store data according to the DL training model; and
compress the stored data based upon a difference between past iterations of performing the DL training model using a compression engine to store the difference between past iterations.
19. The data storage device of claim 18 , wherein the controller is further configured to place data of commands in a specified buffer without involvement of a host device.
20. The data storage device of claim 18 , wherein the performing a DL training model comprises conducting reads and writes according to a pre-defined training schedule.