System and Method of Single Control Plane for Data Ingestion
A method is provided for ingesting datasets into existing data warehouses, which are associated with data warehouse references. A selection of a data warehouse reference into which a dataset is to be loaded is received from a user. The dataset is accessed and its regional and format particulars are determined. Available data warehouses associated with the selected data warehouse reference are determined. Based on the regional and format particulars, a data warehouse is selected from these available data warehouses. A current address of the selected data warehouse is retrieved. The dataset is ingested directly into the selected data warehouse at the current address without further input from the user or knowledge of the current address by the user. A method of migrating between existing data warehouses with ingested data is also provided.
1 . A method of ingesting datasets into existing data warehouses, the warehouses being associated with data warehouse references, comprising:
receiving from a user a selection of a data warehouse reference into which a dataset is to be loaded;
accessing the dataset and determining its regional and format particulars;
determining available data warehouses that are associated with the selected data warehouse reference;
based on the regional and format particulars, selecting a data warehouse from the available data warehouses;
retrieving a current address of the selected data warehouse; and
ingesting the dataset directly into the selected data warehouse at the current address without further input from the user or knowledge of the current address by the user.
2 . The method of claim 1 , further comprising configuring, cleaning or reformatting the dataset prior to loading it into the selected data warehouse.
3 . The method of claim 1 , further comprising making the dataset available for query once ingested based on the data warehouse reference.
4 . The method of claim 3 , wherein the query is tailored for a preferred query format of the data warehouse.
5 . The method of claim 1 , wherein a region of the dataset matches the region of the selected data warehouse.
6 . The method of claim 1 , wherein a language of the dataset matches a language of the selected data warehouse.
7 . The method of claim 1 , wherein a format of the dataset matches a format of the selected data warehouse.
8 . The method of claim 1 , wherein the ingesting is by a process selected from: real time, batchwise, and lambda architecture.
9 . The method of claim 1 , wherein a load marker is set or reset before the ingesting step.
10 . The method of claim 1 , wherein a query marker is set or reset after the ingesting step.
11 . A method of migrating between existing data warehouses having ingested data, the warehouses being associated with warehouse references, comprising:
establishing a first data warehouse associated with a data warehouse reference, and loading data into the first data warehouse;
establishing a second data warehouse;
when migration from the first to the second data warehouse is to occur, switching a load marker from the first to the second data warehouse;
wherein a new revision of the dataset is automatically directable after the migration.
12 . The method of claim 11 , further comprising moving data from the first data warehouse into the second data warehouse.
13 . The method of claim 11 , further comprising copying data from the first data warehouse into the second data warehouse.
14 . The method of claim 11 , further comprising deleting data from the first data warehouse after the migration.
15 . The method of claim 11 , further comprising switching a hot/cold marker on the first data warehouse to cold after the migration.
16 . The method of claim 11 , further comprising switching a hot/cold marker on the second data warehouse to hot after the migration.
17 . The method of claim 11 , wherein the second data warehouse is in a different region than the first data warehouse.
18 . The method of claim 17 , further comprising moving or copying data from the first data warehouse into the second data warehouse, and reformatting the moved or copied data for the region of the second data warehouse.
19 . The method of claim 11 , further comprising automatically ingesting new datasets into the second data warehouse after the migration.
20 . The method of claim 11 , further comprising automatically switching a query marker from the first to the second data warehouse after the migration.
21 . The method of claim 20 , further comprising receiving a query related to a dataset and automatically directing the query to the second data warehouse.
22 . The method of claim 21 , wherein a query counter is updated.
23 . The method of claim 22 , further comprising setting or resetting a hot/cold marker based on the query counter.