Database management based on predicted retrieval of data
Methods and systems for managing data are disclosed. Data may be stored in a database based on a likelihood that the data is to be read from the database. Data may be stored as a reduced-size representation of the data if the data is unlikely to be read from the database. Some requests for data may be associated with timeliness requirements. If data associated with a timeliness requirement is stored as a reduced-size representation, a likelihood of violating the timeliness requirement while reconstructing the data may be increased. Therefore, data that is likely to be requested may be stored in a reconstructed full-size form in a cache of the database. By doing so, the reconstructed full-size forms of the data may be retrieved from the cache and used for providing computer-implemented services in compliance with timeliness requirements for the data.
1 . A method of managing data, the method comprising:
obtaining a portion of the data for storage;
making a determination regarding whether the portion is likely to be read after the portion is stored in a database;
in a first instance of the determination where the portion is likely to be read:
storing the portion in the database;
in a second instance of the determination where the portion is unlikely to be read:
obtaining a reduced-size representation of the portion using a predictive model trained to predict a time-series relationship, wherein the reduced-size representation comprises a plurality identifiers of checkpoints;
storing the reduced-size representation rather than the portion in the database;
populating a cache for the database using the predictive model; and
reconstructing the data based on the checkpoints.
2 . The method of claim 1 , wherein the portion of the data comprises information regarding a time-series relationship.
3 . The method of claim 2 , wherein making the determination comprises:
identifying whether the portion of the data is anomalous with respect to the time-series relationship; and
in an instance of the identifying where the portion of the data is anomalous:
concluding that the portion is likely to be read.
4 . The method of claim 2 , wherein making the determination comprises:
obtaining historical read trends for the database;
comparing the portion of the data to the historical read treads to identify whether the portion of the data would have been read in the past; and
in an instance of the identifying where the portion of the data would have been read in the past:
concluding that the portion is likely to be read.
5 . The method of claim 1 , wherein obtaining the reduced-size representation of the portion comprises:
obtaining, using the predictive model, an inference for the portion of the data;
obtaining a difference between the inference and the portion of the data; and
obtaining the reduced-size representation using the difference.
6 . The method of claim 5 , wherein the inference and the difference are usable to obtain the portion of the data.
7 . The method of claim 6 , wherein the portion of the data comprises information regarding the time-series relationship.
8 . The method of claim 1 , wherein making the determination is based, at least in part, on a set of rules.
9 . The method of claim 8 , wherein the set of the rules is based on portions of the data already stored in the database and the set of the rules indicates:
portions of the data are to be stored in the database as full-size forms of data at regular intervals; and
the portion is to be stored in the database as a full-size form of the data even when the determination concludes that the portion is unlikely to be read based on the regular intervals.
10 . A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing data, the operations comprising:
obtaining a portion of the data for storage;
making a determination regarding whether the portion is likely to be read after the portion is stored in a database;
in a first instance of the determination where the portion is likely to be read:
storing the portion in the database;
in a second instance of the determination where the portion is unlikely to be read:
obtaining a reduced-size representation of the portion using a predictive model trained to predict a time-series relationship, wherein the reduced-size representation comprises a plurality identifiers of checkpoints;
storing the reduced-size representation rather than the portion in the database;
populating a cache for the database using the predictive model; and
reconstructing the data based on the checkpoints.
11 . The non-transitory machine-readable medium of claim 10 , wherein the portion of the data comprises information regarding a time-series relationship.
12 . The non-transitory machine-readable medium of claim 11 , wherein making the determination comprises:
identifying whether the portion of the data is anomalous with respect to the time-series relationship; and
in an instance of the identifying where the portion of the data is anomalous:
concluding that the portion is likely to be read.
13 . The non-transitory machine-readable medium of claim 11 , wherein making the determination comprises:
obtaining historical read trends for the database;
comparing the portion of the data to the historical read treads to identify whether the portion of the data would have been read in the past; and
in an instance of the identifying where the portion of the data would have been read in the past:
concluding that the portion is likely to be read.
14 . The non-transitory machine-readable medium of claim 10 , wherein obtaining the reduced-size representation of the portion comprises:
obtaining, using the predictive model, an inference for the portion of the data;
obtaining a difference between the inference and the portion of the data; and
obtaining the reduced-size representation using the difference.
15 . The non-transitory machine-readable medium of claim 14 , wherein the inference and the difference are usable to obtain the portion of the data.
16 . The non-transitory machine-readable medium of claim 10 , wherein making the determination is based, at least in part, on a set of rules.
17 . A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing data, the operations comprising:
obtaining a portion of the data for storage;
making a determination regarding whether the portion is likely to be read after the portion is stored in a database;
in a first instance of the determination where the portion is likely to be read:
storing the portion in the database;
in a second instance of the determination where the portion is unlikely to be read:
obtaining a reduced-size representation of the portion using a predictive model trained to predict a time-series relationship, wherein the reduced-size representation comprises a plurality identifiers of checkpoints;
storing the reduced-size representation rather than the portion in the database;
populating a cache for the database using the predictive model; and
reconstructing the data based on the checkpoints.
18 . The data processing system of claim 17 , wherein the portion of the data comprises information regarding a time-series relationship.
19 . The data processing system of claim 18 , wherein making the determination comprises:
identifying whether the portion of the data is anomalous with respect to the time-series relationship; and
in an instance of the identifying where the portion of the data is anomalous:
concluding that the portion is likely to be read.
20 . The data processing system of claim 18 , wherein making the determination comprises:
obtaining historical read trends for the database;
comparing the portion of the data to the historical read treads to identify whether the portion of the data would have been read in the past; and
in an instance of the identifying where the portion of the data would have been read in the past:
concluding that the portion is likely to be read.