Intermediate sorting for data set projection builds

1 . A system, comprising:

a plurality of computing devices, respectively comprising at least one processor and a memory, that implement a distributed data store;

wherein the distributed data store is configured to:

receive a request to create a projection of a data set stored across a plurality of data stores according to a different schema;

create the projection of the data set, wherein to create the projection of the data set, the distributed data store is configured to:

access a copy of the data set to sort data obtained from the copy of the data set to include in the projection according to the different schema;

store the sorted data as an intermediate sorted version of the projection of the data set according to the different schema; and

determine a number of portions to be used to store the projection of the data set across the plurality of data stores; and

distribute execution across a plurality of different computing resources to create the projection of the data set from different portions of the number of portions to store at respective ones of the plurality of data stores; and

provide one or more progress indicators of creating the projection of the data set based, at least in part, on the determined number of portions.

2 . The system of claim 1 , wherein to create the projection of the data set, the distributed storage system is configured to:

divide the copy of the data set into a number of chunks to assign to individual computing resources;

cause each computing resource to

read the assigned chunk;

identify data from the chunk to include in the projection according to the different schema;

sort and store the data into an intermediate data store; and

create a globally sorted version of the data of the number of chunks in the intermediate data store.

3 . The system of claim 1 , wherein the distributed storage system is a database service of a provider network, wherein the data set is one or more tables, and wherein the projection is a secondary index.

4 . A method, comprising:

creating a projection of a data set stored across a plurality of data stores according to a different schema, comprising:

accessing a copy of the data set to store an intermediate sorted version of the projection of the data set according to the different schema; and

determining a number of portions to be used to store the projection of the data set across the plurality of data stores; and

distributing execution to create the projection of the data set from different portions of the number of portions to store at respective ones of the plurality of data stores; and

providing one or more progress indicators of creating the projection of the data set based, at least in part, on the determined number of portions.

5 . The method of claim 4 , wherein the one or more progress indicators comprise a completion percentage of creating the projection of the data set.

6 . The method of claim 4 , wherein the one or more progress indicators comprise a time remaining to complete creating the projection of the data set.

7 . The method of claim 4 , wherein distributed execution to create the projection of the data set uses a dedicated bandwidth at the respective ones of the plurality of data stores.

8 . The method of claim 4 , wherein accessing the copy of the data set to: store an intermediate sorted version of the projection of the data set according to the different schema; and determine a number of portions to be used to store the projection of the data set across the plurality of data stores, comprises:

dividing the copy of the data set into a number of chunks to assign to individual computing resources;

causing each computing resource to:

read the assigned chunk;

identify data from the chunk to include in the projection according to the different schema;

sort and store the data into an intermediate data store; and

creating a globally sorted version of the data of the number of chunks in the intermediate data store.

9 . The method of claim 4 , wherein the distributed execution assigns individual computing resources to store data of the projection to one of the plurality data stores.

10 . The method of claim 4 , wherein the copy of the data set is a backup copy created for the data set and corresponding to a point in time.

11 . The method of claim 4 , wherein the data set is one or more tables of a non-relational database system and wherein the projection is a secondary index of the non-relational database system.

12 . One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:

receiving a request to create a projection of a data set stored across a plurality of data stores according to a different schema;

creating a projection of the data set, wherein creating the projection of the data set the program instructions cause the one or more computing devices to implement:

causing access to a copy of the data set to store an intermediate sorted version of the projection of the data set according to the different schema; and

determining a number of portions to be used to store the projection of the data set across the plurality of data stores; and

causing distributed execution to create the projection of the data set from different portions of the number of portions to store at respective ones of the plurality of data stores; and

providing one or more progress indicators of creating the projection of the data set based, at least in part, on the determined number of portions.

13 . The one or more non-transitory, computer-readable storage media of claim 12 , wherein the one or more progress indicators comprise a completion percentage of creating the projection of the data set.

14 . The one or more non-transitory, computer-readable storage media of claim 12 , wherein the one or more progress indicators comprise a time remaining to complete creating the projection of the data set.

15 . The one or more non-transitory, computer-readable storage media of claim 12 , wherein, in creating the projection of the data set, the program instructions cause the one or more computing devices to further implement:

dividing the copy of the data set into a number of chunks to assign to individual computing resources;

causing each computing resource to:

read the assigned chunk;

identify data from the chunk to include in the projection according to the different schema;

sort and store the data into an intermediate data store; and

creating a globally sorted version of the data of the number of chunks in the intermediate data store.

16 . The one or more non-transitory, computer-readable storage media of claim 12 , wherein the copy of the data set is a backup copy created for the data set and corresponding to a point in time.

17 . The one or more non-transitory, computer-readable storage media of claim 12 , wherein the one or more computing devices implement a database service of a provider network, wherein the data set is one or more tables, and wherein the projection is a secondary index.