Virtual data sources of data virtualization-based architecture
Techniques are disclosed relating to managing virtual data sources (VDSs), including creating and using VDSs. A virtual data source manager (VDSM) that is executing on a computer system may receive a VDS creation request to create a particular VDS. The VDS creation request may include information that is usable to identify a data origin of the particular VDS and an ingestion routine. The VDSM may use the ingestion routine to create the particular VDS that includes a bitmap index that is representative of data stored at the origin. In response to a data request corresponding to the origin, the particular VDS may be used to initiate a query to the particular VDS for data without accessing the origin. The VDSM may further maintain various software plugins that are executable to perform corresponding functions in relation to the VDSM and the VDSs managed by the VDSM.
1. A method, comprising:
maintaining, by a computer system, a portion of a virtual data source (VDS) that is based on a corresponding origin, wherein the VDS is distributed among a plurality of computer systems that includes the computer system, and wherein the portion of the VDS includes:
a portion of a bitmap index that is representative of particular data stored at the origin;
translation metadata that is usable to interpret the bitmap index; and
ingestion metadata that specifies a location of the origin and a set of ingest parameters for configuring an ingestion routine to ingest, into the portion of the bitmap index, updates made to the particular data stored at the origin;
based on a particular update to the portion of the bitmap index, computer system:
applying a set of changes to the translation metadata so that the translation metadata is usable to interpret the particular update; and
propagating the set of changes to one or more remaining computer systems of the plurality of computer systems; and
servicing, by the computer system, a request by executing a query against the VDS and not the origin.
2. The method of claim 1 , wherein the origin is an original data source that corresponds to a storage service of a cloud-based platform on a wide area network, and wherein the location corresponds to one or more files maintained by the storage service.
3. The method of claim 2 , wherein the set of ingest parameters specifies:
access credentials that permit the ingestion routine to access the one or more files; and
at least one data format of data objects included in the one or more files to enable the ingestion routine to parse ones of the data objects.
4. The method of claim 1 , wherein the set of ingest parameters specifies:
a storage location at which to store data ingested from the origin for inclusion in the bitmap index, wherein the storage location corresponds to a data store of a storage service that is provided by a cloud-based platform on a wide area network.
5. The method of claim 1 , wherein the VDS includes:
an access list that identifies a set of permissions for accessing data from the bitmap index, wherein a particular one of the set of permissions indicates a particular set of data that can be accessed from the bitmap index by a corresponding entity.
6. The method of claim 1 , wherein the set of ingest parameters specifies:
a mapping of a set of data fields of one or more data objects of the origin to a set of data fields of the bitmap index of the VDS.
7. The method of claim 6 , wherein the set of data fields of the one or more data objects that is defined in the mapping is a subset of all data fields of the one or more data objects.
8. The method of claim 1 , wherein the set of ingest parameters specifies:
a set of transformations for transforming data stored at the origin from a first format to a second, different format for inclusion in the bitmap index.
9. The method of claim 1 , wherein the VDS is associated with a VDS identifier that enables the computer system to identify the VDS separately from other VDSs that are managed by the computer system.
10. The method of claim 1 , wherein the VDS includes:
fingerprint information indicative of the VDS, wherein the fingerprint information specifies a number of rows of the bitmap index, a number of columns of the bitmap index, and a data density of the bitmap index.
11. A non-transitory computer readable medium having program instructions stored thereon that are executable to cause a computer system to perform operations comprising:
maintaining a portion of a virtual data source (VDS) that is based on a corresponding origin, wherein the VDS is distributed among a plurality of computer systems that includes the computer system, and wherein the portion of the VDS includes:
a portion of a bitmap index that is representative of particular data stored at the origin;
translation metadata that is usable to interpret the bitmap index; and
ingestion metadata that specifies a location of the origin and a set of ingest parameters for configuring an ingestion routine to ingest, into the portion of the bitmap index, updates made to the particular data stored at the origin;
based on a particular update to the portion of the bitmap index:
applying a set of changes to the translation metadata so that the translation metadata is usable to interpret the particular update; and
propagating the set of changes to one or more remaining computer systems of the plurality of computer systems; and
servicing a request by executing a query against the VDS and not the origin.
12. The medium of claim 11 , wherein the origin is an original data source that corresponds to a pipeline of a pipeline-based platform that provides a data stream.
13. The medium of claim 12 , wherein the maintaining includes ingesting data from the origin into the bitmap index using the set of ingest parameters, wherein the set of ingest parameters specifies:
a host address corresponding to the pipeline-based platform; and
a pipeline identifier that indicates the pipeline; and
at least one data format of a set of data objects in the pipeline that enables the ingestion routine to parse ones of the set of data objects.
14. The medium of claim 11 , wherein the maintaining includes:
storing data ingested from the origin for inclusion in the bitmap index at a data store included in a compute cluster that implements the VDS, wherein the compute cluster is managed by the computer system.
15. The medium of claim 11 , wherein the VDS is created such that the VDS includes an access list that identifies sets of entities and respective sets of data that can be accessed from the bitmap index by ones of those sets of entities.
16. A system, comprising:
at least one processor; and
a memory having program instructions stored thereon that are executable by the at least one processor to cause the system to perform operations comprising:
maintaining a portion of a virtual data source (VDS) that is based on a corresponding origin, wherein the VDS is distributed among a plurality of systems that includes the system, and wherein the portion of the VDS includes:
a portion of a bitmap index that is representative of particular data stored at the origin;
translation metadata that is usable to interpret the bitmap index; and
ingestion metadata that specifies:
a location of the origin;
a set of parameters for configuring an ingestion routine to ingest, into the portion of the bitmap index, updates made to the particular data stored at the origin; and
based on a particular update to the portion of the bitmap index:
applying a set of changes to the translation metadata so that the translation metadata is usable to interpret the particular update; and
propagating the set of changes to one or more remaining systems of the plurality of systems; and
servicing a request by executing a query against the VDS and not the origin.
17. The system of claim 16 , wherein the origin is an original data source that corresponds to a pipeline of a pipeline-based platform that provides a data stream.
18. The system of claim 17 , wherein the set of other parameters specifies:
a host address corresponding to the pipeline-based platform; and
a pipeline identifier that indicates the pipeline; and
at least one data format of a set of data objects in the pipeline that enables the ingestion routine to parse ones of the set of data objects.
19. The system of claim 16 , wherein the maintaining includes:
storing data ingested from the origin for inclusion in the bitmap index at a data store included in a compute cluster that implements the VDS, wherein the compute cluster is managed by the system.
20. The system of claim 16 , wherein the VDS is created such that the VDS includes an access list that identifies sets of entities and respective sets of data that can be accessed from the bitmap index by ones of those sets of entities.