Retention management for data streams
The described technology is generally directed towards managing data retention policy for stream data stored in a streaming storage system. When a request to truncate a data stream from a certain position (e.g., from a request-specified stream cut) is received, an evaluation is made to determine whether the requested position is within a data retention period as specified by data retention policy. If any data prior to the stream cut position (corresponding to a stream cut time) is within the data retention period, the truncation request is blocked. Otherwise truncation from the stream cut point is allowed to proceed/is performed. Also described is handling automated (e.g., sized based) stream truncation requests with respect to data retention.
1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
receiving a request to truncate a data stream stored via a streaming data storage system, the request associated with a stream cut object maintained for the data stream, wherein the stream cut object is stored via a stream cut auxiliary stream, and wherein the stream cut object corresponds to a stream cut object position within the stream cut auxiliary stream;
obtaining a retention period of selectable retention periods associated with the data stream, wherein the retention period reflects a minimum time for which data of the data stream is to be preserved and after which the data is permitted to be deleted from the data stream;
obtaining a stream cut time associated with the stream cut object, wherein the stream cut time corresponds to an attempt to truncate the data of the data stream occurring earlier than the stream cut time;
determining whether the stream cut time is within the retention period, wherein the stream cut time being within the retention period corresponds to the attempt to truncate the data stream attempting to delete at least some of the data that is not yet permitted to be deleted from the data stream; and
in response to the stream cut time being determined not to be within the retention period, truncating the data stream based on the stream cut object.
2. The system of claim 1 , wherein the determining whether the stream cut time is within the retention period comprises determining a retention end time based on the stream cut time and the retention period, and determining whether the retention end time is later than a current system time.
3. The system of claim 1 , wherein the operations further comprise, in response to the stream cut time being determined to be within the retention period, blocking the request.
4. The system of claim 3 , wherein the request to truncate the data stream is received from a size-based expiration requestor, and wherein the blocking the request corresponds to increasing a capacity footprint of the data stream beyond a pre-specified size limit.
5. The system of claim 4 , wherein the operations further comprise scheduling a delayed truncation operation in response to the increasing the capacity footprint of the data stream.
6. The system of claim 5 , wherein the scheduling the delayed truncation operation comprises scheduling the delayed truncation operation to occur at a stream cut time that is not within the retention time period.
7. The system of claim 1 , wherein the stream cut object is a second stream cut object and the stream cut time value is a second stream cut time value that is after a first stream cut time value of a first stream cut object, stored via the stream cut auxiliary stream, earlier in time than the second stream cut object, and wherein the operations further comprise,
in response to determining that the second stream cut time value is within the retention period, selecting the first stream cut object, and determining whether the first stream cut time value is within the retention period, and
in response to determining that the first stream cut time is not within the retention period, truncating the data stream based on the first stream cut object.
8. The system of claim 1 , wherein the operations further comprise maintaining the steam cut auxiliary stream comprising the stream cut object, wherein the stream cut object further corresponds to a stream cut event creation time.
9. The system of claim 8 , wherein the operations further comprise, in response to the stream cut time being determined not to be within the retention period, truncating the auxiliary stream based on the stream cut object.
10. A method, comprising:
receiving, by a system comprising a processor, a request to truncate a data stream stored via a streaming data storage system, the request associated with a stream cut object maintained for the data stream, wherein the stream cut object is stored via a stream cut auxiliary stream, and wherein the stream cut object corresponds to a stream cut object position within the stream cut auxiliary stream;
determining, by the system, a retention period of retention periods associated with the data stream, wherein the retention period reflects a time for which data of the data stream is to be preserved and after which the data is permitted to be deleted from the data stream;
determining, by the system, a stream cut time associated with the stream cut object, wherein the stream cut time corresponds to a truncation operation to the data of the data stream occurring earlier than the stream cut time;
determining, by the system, whether the stream cut time is within the retention period, wherein the stream cut time being within the retention period corresponds to the truncation operation to the data stream attempting to delete at least some of the data that is not yet permitted to be deleted from the data stream; and
in response to the stream cut time being determined, by the system, not to be within the retention period, causing the truncation operation to the data stream to occur based on the stream cut object.
11. The method of claim 10 , wherein the determining that the stream cut time is not comprised in the retention period is based on the retention period having an end time is later than a current system time.
12. The method of claim 10 , wherein the determining that the stream cut time is not comprised in the retention period is based on the retention period having an end time is earlier than a current system time.
13. The method of claim 10 , further comprising, in response to the stream cut time being determined to be within the retention period, blocking, by the system, the request to prevent truncation of the data stream prior to the retention period.
14. The method of claim 10 , wherein the receiving the request comprises receiving the request from a size-based expiration requestor, and wherein the request corresponds to increasing a capacity footprint of the data stream beyond a pre-specified size limit.
15. The method of claim 14 , further comprising scheduling, by the system, a delayed truncation operation in response to the increasing the capacity footprint of the data stream.
16. The method of claim 15 , wherein the scheduling results in causing the delayed truncation operation to occur at a stream cut time that is not within the retention time period.
17. A machine-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
receiving a request to truncate a data stream stored via a streaming data storage system, the request associated with a stream cut object maintained for the data stream, wherein the stream cut object is stored via a stream cut auxiliary stream, and wherein the stream cut object corresponds to a stream cut object position within the stream cut auxiliary stream;
receiving a retention period of selectable retention periods associated with the data stream, wherein the retention period reflects a minimum time for which data of the data stream is to be preserved and after which the data is permitted to be deleted from the data stream;
receiving a stream cut time associated with the stream cut object, wherein the stream cut time corresponds to an attempt to truncate the data of the data stream occurring earlier than the stream cut time;
in response to determining that the stream cut time is not within the retention period, wherein the stream cut time being instead determined to be within the retention period corresponds to the attempt to truncate the data stream attempting to delete at least some of the data that is not yet permitted to be deleted from the data stream, truncating the data stream based on the stream cut object.
18. The machine-readable storage medium of claim 17 , wherein the data stream is a first data stream, and wherein the selectable retention periods are stored via a second data stream of the streaming data storage system.
19. The machine-readable storage medium of claim 18 , wherein the first data stream is a different data stream than the second data stream.
20. The machine-readable storage medium of claim 18 , wherein the first data stream is a same data stream as the second data stream.