Maintaining data stream history for generating materialized views
Materialized views may be generated at a managed materialized view platform that accepts data streams as a source. When updating a materialized view with a data stream as a source, a stored portion of the stream is used to maintain a history of changes to be made. When a join operation needs to utilize the data stream source to determine an update to the materialized view, the data stream is preserved to complete the join and update the materialized view.
1. A system, comprising:
one or more processors; and
a memory, that stores program instructions that, when executed by the at least one processor, cause the one or more processors to implement a service of a provider network, configured to:
receive a request to create a materialized view stored at a target data store according to a definition that specifies a join key to identify matches between a first data source and a second data source, wherein application of the join key joins first data from the first data source that is a data stream with second data from the second data source;
create the materialized view at the target data store that joins the first data from the data stream with the second data from the second data source according to the definition of the materialized view that specifies the join key; and
maintain the materialized view to include further data obtained from the data stream joined with additional data obtained the second data source according to the definition of the materialized view that specifies the join key.
2. The system of claim 1 , wherein the second data source is a second data stream.
3. The system of claim 1 , wherein the second data source is a data store.
4. The system of claim 1 , wherein the materialized view is further created at a second target data store in addition to the target data store, wherein the target data store and the second target data store are selected according to a list of targets offered by the service of the provider network.
5. The system of claim 1 , wherein to maintain the materialized view to include the further data obtained from the data stream joined with the additional data obtained from the second data source, the service of the provider network is configured to access a stored portion of the data stream that includes the further data to determine a new version of the materialized view.
6. The system of claim 5 , wherein the service of the provider network is further configured to:
obtain the further data from the data stream;
append the further data as one or more entries in an append-only table maintained for the data stream, wherein the append-only table provides the stored portion of the data stream.
7. The system of claim 1 , wherein the target data store is another service offered by the provider network.
8. A method, comprising:
receiving, by a service of a provider network, a request to create a materialized view stored at a target data store according to a definition that specifies a join key to identify matches between a first data source and a second data source, wherein application of the join key joins first data from the first data source that is a data stream with second data from the second data source;
creating, by the service of the provider network, the materialized view at the target data store that joins the first data from the data stream with the second data from the second data source according to the definition of the materialized view that specifies the join key; and
maintaining, by the service of the provider network, the materialized view to include further data obtained from the data stream joined with additional data obtained the second data source according to the definition of the materialized view that specifies the join key.
9. The method of claim 8 , wherein the second data source is a second data stream.
10. The method of claim 8 , wherein the second data source is a data store.
11. The method of claim 8 , wherein the materialized view is further created at a second target data store in addition to the target data store, wherein the target data store and the second target data store are selected according to a list of targets offered by the service of the provider network.
12. The method of claim 8 , wherein maintaining the materialized view to include the further data obtained from the data stream joined with the additional data obtained from the second data source comprises accessing a stored portion of the data stream that includes the further data to determine a new version of the materialized view.
13. The method of claim 12 , further comprising:
obtaining, by the service of the provider network, the further data from the data stream;
appending, by the service of the provider network, the further data as one or more entries in an append-only table maintained for the data stream, wherein the append-only table provides the stored portion of the data stream.
14. The method of claim 8 , wherein the target data store is another service offered by the provider network.
15. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement:
receiving, by a service of a provider network, a request to create a materialized view stored at a target data store according to a definition that specifies a join key to identify matches between a first data source and a second data source, wherein application of the join key joins first data from the first data source that is a data stream with second data from the second data source;
creating, by the service of the provider network, the materialized view at the target data store that joins the first data from the data stream with the second data from the second data source according to the definition of the materialized view that specifies the join key; and
maintaining, by the service of the provider network, the materialized view to include further data obtained from the data stream joined with additional data obtained the second data source according to the definition of the materialized view that specifies the join key.
16. The one or more non-transitory computer-readable storage media of claim 15 , wherein the second data source is a second data stream.
17. The one or more non-transitory computer-readable storage media of claim 15 , wherein the second data source is a data store.
18. The one or more non-transitory computer-readable storage media of claim 15 , wherein the materialized view is further created at a second target data store in addition to the target data store, wherein the target data store and the second target data store are selected according to a list of targets offered by the service of the provider network.
19. The one or more non-transitory computer-readable storage media of claim 15 , wherein, in maintaining the materialized view to include the further data obtained from the data stream joined with the additional data obtained from the second data source, the program instructions cause the one or more computing devices to implement accessing a stored portion of the data stream that includes the further data to determine a new version of the materialized view.
20. The one or more non-transitory computer-readable storage media of claim 19 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:
obtaining, by the service of the provider network, the further data from the data stream;
appending, by the service of the provider network, the further data as one or more entries in an append-only table maintained for the data stream, wherein the append-only table provides the stored portion of the data stream.