Data system with flush views
A data storage system includes a head node and mass storage devices. The head node is configured to flush data stored in a storage of the head node to a set of the mass storage devices of the data storage system. A head node may flush both current version data and point-in-time version data to the set of mass storage devices. Also, the data storage system maintains an index that indicates storage locations of data for particular portions of a volume before and after the data is flushed to the set of mass storage devices. In some embodiments an index includes a current version reference for a volume or portion of a volume and one or more point-in-time snapshot references.
1. A system, comprising:
one or more computing devices configured to implement a block-based storage service configured to:
provide block-based storage volumes configured to be attached to respective compute instances provided by a virtual compute service of a provider network;
receive, from a client of the provider network, via an application programmatic interface (API), a request to implement snapshotting of a given one of the block-based storage volumes, wherein the request specifies:
a frequency-based criteria for generating one or more snapshots of a first portion of the given one of the block-based storage volumes, wherein each of the one or more snapshots represents a state of the first portion of the given one of the block-based storage volumes at a different fixed point-in-time; and
a write-based criteria for generating one or more other snapshots of a second portion of the given one of the block-based storage volumes, wherein each of the other one or more snapshots represents a state of the second portion of the given one of the block-based storage volumes at a different fixed point-in-time; and
implement snapshotting for the first portion of the given one of the block-based storage volumes and the second portion of the given one of the block-based storage volumes in accordance with the frequency-based criteria and the write-based criteria indicated by the client such that:
one or more snapshots are generated for the first portion of the given one of the block-based storage volumes based on the frequency-based criteria; and
one or more other snapshots are generated for the second portion of the given one of the block-based storage volumes based on the write-based criteria.
2. The system of claim 1 , wherein the block-based service is further configured to:
receive another request to implement snapshotting at a different given one of the block-based storage volumes, wherein the another request specifies:
a first criteria for generating one or more snapshots at a first portion of the different given one of the block-based storage volumes; and
a second criteria for generating one or more other snapshots at a second portion of the different given one of the block-based storage volumes;
wherein the first criteria is a first time-based frequency and the second criteria is a different time-based frequency,
wherein snapshotting is implemented such that snapshots are generated for the first portion of the different given one of the block-based storage volumes and the second portion of the different given one of the block-based storage volumes at different frequencies.
3. The system of claim 1 , wherein the request further specifies an offset range of the given one of the block-based storage volumes for which the write-based criteria for generating snapshots is to be applied.
4. The system of claim 3 , wherein snapshotting is implemented such that snapshots are generated for the second portion of the given one of the block-based storage volumes when a write is performed affecting the specified offset range corresponding to the first portion of the given one of the block-based storage volumes.
5. The system of claim 4 , wherein the frequency-based criteria is a time-based frequency,
wherein snapshotting is further implemented such that snapshots are generated for the first portion of the given one of the block-based storage volumes according to the time-based frequency.
6. The system of claim 4 , wherein the request further specifies a time range for which a write-based frequency is to be used to implement snapshotting for the second portion of the given one of the block-based storage volumes,
wherein snapshotting is implemented such that snapshots are generated for the second portion of the given one of the block-based storage volumes when a write is performed affecting the specified offset range corresponding to the second portion of the given one of the block-based storage volumes during the specified time range, and wherein snapshots are generated for the second portion of the given one of the block-based storage volumes according to a time-based frequency outside of the specified time range.
7. The system of claim 1 , wherein the one or more computing devices that implement the block-based storage comprise:
a log storage and an associated index indicating storage locations for volume data stored in the log storage, and
wherein to generate a snapshot of the given one of the block-based storage volumes, the one or more computing devices that implement the block-based storage are configured to:
update or add a reference for the given volume to the index such that the index includes a current version reference for the given one of the block-based storage volumes and a snapshot reference for the given one of the block-based storage volumes.
8. The system of claim 7 , wherein the one or more computing devices that implement the block-based storage are further configured to:
receive additional writes for the given volume;
store additional data included in the additional writes for at least one of the first portion of the given one of the block-based storage volumes or the second portion of the given one of the block-based storage volumes subsequent to generating one or more snapshots for the first portion of the given one of the block-based storage volumes or the second portion of the given one of the block-based storage volumes;
receive a request for a snapshot copy of the given one of the block-based storage volumes at a point-in-time of one of the one or more generated snapshots; and
generate a snapshot copy of the first or second portion of the given one of the block-based storage volumes at the requested point in time according to data stored at locations indicated by a given snapshot reference for the requested point in time snapshot.
9. The system of claim 8 , wherein the one or more computing devices that implement the block-based storage are further configured to:
cause at least some of the data and at least some of the additional data stored in the log storage to be written from the log storage to a set of mass storage devices; and
update the index for the log storage such that the current version reference and the snapshot version reference indicate locations in the set of mass storage devices at which the at least some data or additional data for the given one of the block-based storage volumes is stored.
10. The system of claim 8 , wherein the one or more computing devices that implement the block-based storage are further configured to:
export snapshot copies of the first portion of the given one of the block-based storage volumes or the second portion of the given one of the block-based storage volumes to an object-based storage system of the provider network.
11. A method, comprising:
providing a block-based storage volume;
receiving, via an application programmatic interface (API), a request to implement snapshotting for the block-based storage volume, wherein the request specifies:
a frequency-based snapshotting criteria for generating snapshots of a first portion of the block-based storage volume, wherein each of the snapshots represents a state of the first portion of the block-based storage volume at a different fixed point-in-time; and
a write-based snapshotting criteria for generating other snapshots of a second portion of the block-based storage volume, wherein each of the other snapshots represents a state of the second portion of the block-based storage volume at a different fixed point-in-time; and
implementing snapshotting of the block-based storage volume in accordance with the frequency-based snapshotting criteria and the write-based snapshotting criteria.
12. The method of claim 11 , further comprising:
receiving another request to implement snapshotting for a different block-based storage volume, wherein the another request specifies:
a first criteria for generating one or more snapshots at a first portion of the different block-based storage volume; and
a second criteria for generating one or more other snapshots at a second portion of the different block-based storage volume;
wherein the first snapshotting criteria is a first time-based frequency and the second snapshotting criteria is a different time-based frequency,
wherein snapshotting is implemented such that snapshots are generated for the first and second portions of the different block-based storage volume at different frequencies.
13. The method of claim 11 , wherein the frequency-based snapshotting criteria is that snapshots are to be generated for the first portion of the block-based storage volume and the write-based snapshotting criteria is that snapshots are not to be generated for the second portion of the block-based storage volume.
14. The method of claim 11 , wherein the frequency-based snapshotting criteria is that snapshots are to be generated for the first portion of the block-based storage volume according to a time-based frequency and wherein the write-based snapshotting criteria is that snapshots are to be generated for the second portion of the block-based storage volume according to a write-based frequency.
15. The method of claim 14 , wherein the write-based snapshotting criteria specifies using the write-based frequency for a specified time range and otherwise using a time-based frequency when generating snapshots for the second portion of the block-based storage volume.
16. The method of claim 11 , wherein the request specifies an offset range for the first portion of the block-based storage volume or the second portion of the block-based storage volume.
17. One or more non-transitory, computer-readable, media storing program instructions, that when executed on or across one or more processors, cause the one or more processors to:
provide a block-based storage volume;
receive, via an application programmatic interface (API), a request to implement snapshotting for the block-based storage volume, wherein the request specifies:
a frequency-based snapshotting criteria for generating snapshots of a first portion of the block-based storage volume, wherein each of the snapshots represents a state of the first portion of the block-based storage volume at a different fixed point-in-time; and
a write-based snapshotting criteria for generating other snapshots of a second portion of the block-based storage volume, wherein each of the other snapshots represents a state of the second portion of the block-based storage volume at a different fixed point-in-time; and
implement snapshotting of the block-based storage volume in accordance with the frequency-based snapshotting criteria and the write-based snapshotting criteria.
18. The one or more non-transitory, computer-readable, media of claim 17 , wherein the program instructions, when executed on or across the one or more processors, further cause the one or more processors to:
receive another request to implement snapshotting for a different block-based storage volume, wherein the another request specifies:
a first snapshotting criteria for generating one or more snapshots at a first portion of the different block-based storage volume; and
a second snapshotting criteria for generating one or more other snapshots at a second portion of the different block-based storage volume;
wherein the first snapshotting criteria and the second snapshotting criteria specify different frequencies at which snapshots are to be generated for the first and second portions of the different block-based storage volume.
19. The one or more non-transitory, computer-readable, media of claim 18 , wherein the different frequencies are different time-based frequencies.
20. The one or more non-transitory, computer-readable, media of claim 17 , wherein the write-based snapshotting criteria specifies a write-based frequency.