Persisting large volumes of data in an efficient unobtrusive manner
View Patent ↗Embodiments provide a data persisting mechanism that allows for efficient, unobtrusive persisting of large volumes of data while optimizing the use of system resources by the persisting process. In an embodiment, the persisting process includes a self-tuning algorithm that constantly monitors persistence performance and that adjusts persistence time to maintain performance within user-defined criteria. From one aspect, this allows the persisting process to seamlessly adapt to changes in system environment (speeding up persistence during times of low processor usage and slowing down persistence during times of high processor usage) and to reduce or eliminate CPU spikes caused by persisting process. From another aspect, the persisting process results in the data being persisted as quickly as possible given the system constraints, thereby minimizing the possibility of data loss.
1. A system, comprising:
a non-transitory computer readable storage medium; and
a processor, coupled to the non-transitory computer readable storage medium, configured to execute an application, wherein the application is configured to:
determine, based on a current performance level of the system, a first persistence data rate for persisting a plurality of objects to the storage, wherein the plurality of objects that are to be persisted in each persistence cycle time are divided into a plurality of object segments of equal size, wherein the persistence cycle time is divided into a plurality of time segments, and wherein each object segment is persisted during each time segment;
if a persistence time required to persist the plurality of objects to the storage at the first persistence data rate is within a persistence cycle time, persist the plurality of objects to the storage at the first persistence data rate; and
if the persistence time required to persist the plurality of objects to the storage at the first persistence data rate exceeds the persistence cycle time,
increase the first persistence data rate to generate a second persistence data rate, wherein said increasing the first persistent rate comprises increasing a segment size of one or more object segments, decreasing a segment length of one or more time segments, or a combination thereof, and
persist the plurality of objects to the storage at the second persistence data rate.
2. The system of claim 1 , wherein the application is a performance monitoring application, the application further configured to:
monitor a plurality of system components to generate performance metrics data; and
process the performance metrics data to generate the plurality of objects.
3. The system of claim 1 , wherein the application is further configured to:
increase a previous persistence data rate to determine the first persistence data rate when the current performance level exceeds a desired performance level; and
decrease a previous persistence data rate to determine the first persistence data rate when the current performance level is below the desired performance level.
4. The system of claim 3 , wherein the current performance level exceeds the desired performance level, and wherein a first persistence time incurred to persist the plurality of objects to the storage at the first persistence data rate is lower than a previous persistence time incurred to persist an equal number of objects to the storage at the previous persistence data rate in a previous persistence cycle.
5. The system of claim 3 , wherein the application is further configured to:
increase a current segment size, the current segment size indicating a number of objects to persist per a current segment length, to increase the previous persistence data rate; and
decrease the current segment size to decrease the previous persistence data rate.
6. The system of claim 5 , wherein the application is further configured to:
determine a required number of segments required to persist the plurality of objects within the persistence cycle time based on the current segment size; and
compare the required number of segments to a current number of segments in the persistence cycle time.
7. The system of claim 6 , wherein the required number of segments exceeds the current number of segments, the application is further configured to:
decrease the current segment length to increase the first persistence data rate to generate the second persistence data rate.
8. A method for persisting data to a storage responsive to system performance, comprising:
determining, based on a current system performance level, a first persistence data rate for persisting a plurality of objects to the storage, wherein the plurality of objects that are to be persisted in each persistence cycle time are divided into a plurality of object segments of equal size, wherein the persistence cycle time is divided into a plurality of time segments, and wherein each object segment is persisted during each time segment;
if a persistence time required to persist the plurality of objects to the storage at the first persistence data rate is within a persistence cycle time, persisting the plurality of objects to the storage at the first persistence data rate; and
if the persistence time required to persist the plurality of objects to the storage at the first persistence data rate exceeds the persistence cycle time,
increasing the first persistence data rate to generate a second persistence data rate, wherein said increasing the first persistent rate comprises increasing a segment size of one or more object segments, decreasing a segment length of one or more time segments, or a combination thereof, and
persisting the plurality of objects to the storage at the second persistence data rate.
9. The method of claim 8 , further comprising:
increasing a previous persistence data rate to determine the first persistence data rate when the current system performance level exceeds a desired performance level; and
decreasing the previous persistence data rate to determine the first persistence data rate when the current system performance level is below the desired performance level.
10. The method of claim 9 , wherein the current system performance level exceeds the desired performance level, and wherein a first persistence time incurred to persist the plurality of objects to the storage at the first persistence data rate is lower than a previous persistence time incurred to persist an equal number of objects to the storage at the previous persistence data rate in a previous persistence cycle.
11. The method of claim 9 , wherein increasing the previous persistence data rate comprises increasing a current segment size, the current segment size indicating a number of objects to persist per a current segment length, and wherein decreasing the previous persistence data rate comprises decreasing the current segment size.
12. The method of claim 11 , wherein the current system performance level is below the desired performance level, the method further comprising:
determining a required number of segments required to persist the plurality of objects within the persistence cycle time based on the current segment size; and
comparing the required number of segments to a current number of segments in the persistence cycle time.
13. The method of claim 12 , wherein the required number of segments exceeds the current number of segments, and wherein increasing the first persistence data rate to generate the second persistence data rate comprises decreasing the current segment length.
14. A computer program product comprising a non-transitory computer readable medium having control logic stored therein, the control logic, when executed by a processor, causes the processor to perform a method for persisting data to a storage responsive to system performance, the method comprising:
determining, based on a current system performance level, a first persistence data rate for persisting a plurality of objects to the storage, wherein the plurality of objects that are to be persisted in each persistence cycle time are divided into a plurality of object segments of equal size, wherein the persistence cycle time is divided into a plurality of time segments, and wherein each object segment is persisted during a corresponding time segment;
if a persistence time required to persist the plurality of objects to the storage at the first persistence data rate is within a persistence cycle time, persisting the plurality of objects to the storage at the first persistence data rate; and
if the persistence time required to persist the plurality of objects to the storage at the first persistence data rate exceeds the persistence cycle time,
increasing the first persistence data rate to generate a second persistence data rate, wherein said increasing the first persistent rate comprises increasing a segment size of one or more object segments, decreasing a segment length of one or more time segments, or a combination thereof, and
persisting the plurality of objects to the storage at the second persistence data rate.
15. The computer program product of claim 14 , the method further comprising:
increasing a previous persistence data rate to determine the first persistence data rate when the current system performance level exceeds a desired performance level; and
decreasing the previous persistence data rate to determine the first persistence data rate when the current system performance level is below the desired performance level.
16. The computer program product of claim 15 , wherein the current performance level exceeds the desired performance level, and wherein a first persistence time incurred to persist the plurality of objects to the storage at the first persistence data rate is lower than a previous persistence time incurred to persist an equal number of objects to the storage at the previous persistence data rate in a previous persistence cycle.
17. The computer program product of claim 15 , wherein increasing the previous persistence data rate comprises increasing a current segment size, the current segment size indicating a number of objects to persist per a current segment length, and wherein decreasing the previous persistence data rate comprises decreasing the current segment size.
18. The computer program product of claim 17 , wherein the current system performance level is below the desired performance level, the method further comprising:
determining a required number of segments required to persist the plurality of objects within the persistence cycle time based on the current segment size; and
comparing the required number of segments to a current number of segments in the persistence cycle time.
19. The computer program product of claim 18 , wherein the required number of segments exceeds the current number of segments, and wherein increasing the first persistence data rate to generate the second persistence data rate comprises decreasing the current segment length.
20. The computer program product of claim 14 , the method further comprising:
monitor a plurality of system components to generate performance metrics data; and
processing the performance metrics data to generate the plurality of objects.