Compressing old machine learning dataset data for use by a human

1 . A data management system for reducing storage resources while maintaining usability of stored data for (i) machine-learning learning and inference during an active phase and (ii) subsequent verification by a human user during an archival phase, comprising:

a storage device configured to store data usable for learning and inference in a machine-learning model;

a processor connected to the storage device;

a model-use compressor/decompressor configured to compress and decompress data into a representation usable for learning and inference by the machine-learning model; and

a human verification compressor/decompressor configured to compress and decompress data into a representation presentable to at least one human user for verification;

wherein the processor is configured to:

(a) acquire data from a data generation source;

(b) compress the acquired data using the model-use compressor/decompressor to generate first compressed data stored in the storage device for learning and inference by the machine-learning model;

(c) identify, among the first compressed data stored in the storage device, specified data that is no longer used for learning and inference in the machine learning model;

(d) generate second compressed data by recompressing the specified data using the human-verification compressor/decompressor; and

(e) store the second compressed data in the storage device in association with the specified data,

wherein the human-verification compressor/decompressor, when decompressing the second compressed data, outputs a human-verifiable reconstruction of the specified data that is usable by the human user to perform said verification, and

wherein storing the second compressed data in place of, or in addition to, the first compressed data reduces an amount of storage required for retaining the specified data while maintaining usability of the specified data for said subsequent verification by the human user.

2 . The data management system of claim 1 , further comprising:

wherein the first compressed data stored in the storage device is data obtained by compressing the data acquired from the data generation source using the model-use compressor/decompressor.

3 . The data management system of claim 1 , wherein the

human-verification compressor/decompressor comprises a neural network trained using a loss function that emphasizes compression ratio while satisfying a human-verification constraint that the human-verifiable reconstruction remains suitable for verification by the human user.

4 . The data management system of claim 3 , wherein:

(a) the data generated by the data generation source includes an image;

(b) the human-verification constraint is defined by an image-quality evaluation index based on a difficulty, by the human user, of distinguishing (i) an original image from (ii) a reconstructed image obtained by decompressing the second compressed data; and

(c) the image-quality evaluation index is used in training the neural network of the human-verification compressor/decompressor.

5 . The data management system of claim 1 , wherein the processor identifies the specified data as no longer used for learning and inference based on at least one retention condition comprising (i) elapsed time since acquisition, (ii) a quantity of newly acquired data exceeding a threshold, (iii) completion of a training update of the machine-learning model, or (iv) expiration of a training dataset window used for the learning.

6 . The data management system of claim 1 , wherein generating the second compressed data comprises decompressing the first compressed data using the model-use compressor/decompressor to obtain an intermediate reconstruction and compressing the intermediate reconstruction using the human-verification compressor/decompressor.

7 . The data management system of claim 1 , wherein the processor, upon storing the second compressed data, deletes or deprecates storage of the first compressed data corresponding to the specified data to reduce the amount of storage required for retaining the specified data.

8 . The data management system of claim 1 , wherein the storage device comprises a first storage tier for the first compressed data and a second storage tier for the second compressed data, and wherein the second storage tier has lower cost per unit storage than the first storage tier.

9 . The data management system of claim 1 , wherein the processor stores, in association with the second compressed data, provenance metadata identifying at least one of: (i) a time at which recompression occurred, (ii) a version identifier of the human-verification compressor/decompressor, (iii) a version identifier of the machine-learning model, or (iv) the retention condition that caused the recompression.

10 . The data management system of claim 1 , wherein the processor stores, in association with the specified data, a pointer or mapping from the first compressed data to the second compressed data, and wherein the pointer or mapping enables retrieval of the second compressed data in response to a request to access the specified data.

11 . The data management system of claim 1 , wherein access to decompress the second compressed data using the human-verification compressor/decompressor is restricted by an access control policy such that only authorized users can obtain the human-verifiable reconstruction.

12 . The data management system of claim 1 , wherein the processor computes and stores, in association with the second compressed data, an integrity value comprising a checksum or cryptographic hash, and verifies the integrity value prior to decompressing the second compressed data.

13 . The data management system of claim 1 , wherein the processor determines a target compression ratio for the human-verification compressor/decompressor based on at least one of: (i) an expected archival retention duration, (ii) an access frequency estimate for verification, or (iii) a storage budget, and generates the second compressed data using the target compression ratio.

14 . The data management system of claim 1 , wherein the human-verification compressor/decompressor is configured to produce the human-verifiable reconstruction at a resolution or quality level selectable by the human user, and wherein the selection affects decompression parameters without changing the second compressed data.

15 . The data management system of claim 1 , wherein the processor maintains an audit log recording at least one of: (i) identification of the specified data, (ii) generation of the second compressed data, (iii) deletion or deprecation of the first compressed data, or (iv) access to the human-verifiable reconstruction by the human user.

16 . The data management system of claim 1 , wherein the processor performs the recompressing of the specified data to generate the second compressed data as a background task scheduled based on available compute resources or I/O bandwidth.

17 . The data management system of claim 1 , wherein the processor selectively refrains from recompressing a data item that would otherwise be identified as no longer used for learning and inference when the data item is associated with a hold flag indicating litigation hold, regulatory retention, or ongoing investigation.

18 . The data management system of claim 1 , wherein the model-use compressor/decompressor comprises a neural network trained using a loss function that emphasizes a period of time required for compression and decompression while satisfying a model-performance constraint for learning and inference by the machine-learning model.

19 . The data management system of claim 18 , wherein an image-quality evaluation index used in training the neural network of the model-use compressor/decompressor comprises an accuracy of the machine-learning model measured when performing inference using images produced by decompressing the first compressed data.

20 . A data management method for reducing storage resources while maintaining usability of retained data for subsequent verification by a human user after completion of a machine-learning learning and inference phase, in a data management system including a processor connected to a storage device that stores data usable for learning and inference in a machine-learning model, the data management system including a model-use compressor/decompressor and a human verification compressor/decompressor, the method comprising:

(a) acquiring data from a data generation source;

(b) compressing the acquired data using the model-use compressor/decompressor to specify data generate first compressed data stored in the storage device for learning and inference by the machine-learning model;

(c) specifying data, among the first compressed data stored in the storage device, that is no longer used for learning and inference in the machine-learning model; and

(d) generating second compressed data by recompressing the specified data using the human-verification compressor/decompressor, such that decompression of the second compressed data produces a human-verifiable reconstruction usable by a human user to perform verification,

wherein storing the second compressed data reduces an amount of storage required for retaining the specified data for said subsequent verification.