Software plugins of data virtualization-based architecture
Techniques are disclosed relating to managing virtual data sources (VDSs), including creating and using VDSs. A virtual data source manager (VDSM) that is executing on a computer system may receive a VDS creation request to create a particular VDS. The VDS creation request may include information that is usable to identify a data origin of the particular VDS and an ingestion routine. The VDSM may use the ingestion routine to create the particular VDS that includes a bitmap index that is representative of data stored at the origin. In response to a data request corresponding to the origin, the particular VDS may be used to initiate a query to the particular VDS for data without accessing the origin. The VDSM may further maintain various software plugins that are executable to perform corresponding functions in relation to the VDSM and the VDSs managed by the VDSM.
1. A method, comprising:
maintaining, by a virtual data source manager (VDSM) that is executing on a computer system, a library of software plugins that are executable to perform functions relating to virtual data sources (VDSs);
receiving, by the VDSM, a request to create a VDS based on an origin; and
creating, by the VDSM based on the request, a particular VDS having a bitmap index representative of data at the origin, wherein the creating includes:
receiving plugin information that specifies a set of software plugins included in the library of software plugins;
configuring the set of software plugins to be able to perform functions relating to the particular VDS;
receiving, by the VDSM, a request for data included in the bitmap index, wherein the request for data specifies one or more queries in a first format; and
spawning, by the VDSM, a consumption software plugin from the set of software plugins to process the request for data, wherein the consumption software plugin is capable of translating the one or more queries from the first format to a second, different format that can be processed against the bitmap index.
2. The method of claim 1 , wherein the set of software plugins includes an ingest software plugin that is capable of accessing data of the origin and converting that data from a first format used at the origin to a second, different format that permits that data to be stored in the bitmap index.
3. The method of claim 2 , wherein the ingest software plugin is capable of accessing data of the origin by ingesting data from a change data capture feed that identifies data changes to the origin.
4. The method of claim 2 , wherein the ingest software plugin is capable of accessing data of the origin by identifying data changes in a file system of the origin and propagating the data changes into the bitmap index.
5. The method of claim 2 , wherein the set of software plugins includes a second ingest software plugin that is capable of ingesting ingested data into the bitmap index that is produced by the ingest software plugin.
6. The method of claim 2 , wherein the origin is located externally to the particular VDS, and wherein the creating includes:
causing, by the VDSM, the ingest software plugin to be executed on a computer system of a local-area network that includes the origin.
7. The method of claim 6 , wherein the ingest software plugin is capable of compressing data ingested from the origin before sending that ingested data from the local-area network that includes the origin to a different local-area network that includes the particular VDS.
8. The method of claim 1 , wherein the set of software plugins includes a transformation software plugin that is capable of performing a set of transformations on values of one or more data fields of the origin to generate values for a particular data field of the bitmap index that is not included in the data at the origin.
9. The method of claim 1 , further comprising:
causing, by the VDSM, an interface to be displayed to a user via a user device, wherein the interface presents software plugins from the library of software plugins, and wherein the plugin information corresponds to a selection of the set of software plugins by the user via the interface.
10. A non-transitory computer readable medium having program instructions stored thereon that are executable to cause a computer system to implement a virtual data source manager (VDSM) capable of performing operations comprising:
maintaining a library of software plugins that are executable to perform functions relating to virtual data sources (VDSs);
receiving a request to create a VDS based on an origin;
creating, based on the request, a particular VDS having a bitmap index representative of data at the origin, wherein the creating includes:
receiving plugin information that specifies a set of software plugins included in the library of software plugins; and
configuring the set of software plugins to be able to perform functions relating to the particular VDS;
receiving a request for data included in the bitmap index, wherein the request for data specifies one or more queries in a first format; and
spawning a consumption software plugin from the set of software plugins to process the request for data, wherein the consumption software plugin is capable of translating the one or more queries from the first format to a second, different format that can be processed against the bitmap index.
11. The medium of claim 10 , wherein the operations further comprise:
spawning a consumption software plugin included in the set of software plugins that is capable of performing a set of data transformations on data being sent from the particular VDS to a destination entity.
12. A system, comprising:
at least one processor; and
memory having program instructions stored thereon that are executable by the at least one processor to cause the system to perform operations comprising:
maintaining a library of software plugins that are executable to perform functions relating to virtual data sources (VDSs);
receiving a request to create a VDS based on an origin;
creating, based on the request, a particular VDS having a bitmap index representative of data at the origin, wherein the creating includes:
receiving plugin information that specifies a set of software plugins included in the library of software plugins; and
configuring the set of software plugins to be able to perform functions relating to the particular VDS; and
receiving a request for data included in the bitmap index, wherein the request for data specifies one or more queries in a first format; and
spawning a consumption software plugin from the set of software plugins to process the request for data, wherein the consumption software plugin is capable of translating the one or more queries from the first format to a second, different format that can be processed against the bitmap index.
13. The system of claim 12 , wherein the set of software plugins includes a second consumption software plugin that is capable of performing a data integrity operation to verify that data included in the bitmap index is representative of data included in the origin.
14. The system of claim 12 , wherein the set of software plugins includes a second consumption software plugin that is capable of executing one or more machine learning models against data included in the bitmap index.
15. The system of claim 12 , wherein the set of software plugins includes a security software plugin that is capable of establishing a secure connection for sending data from the particular VDS to an endpoint system.
16. The system of claim 12 , wherein the set of software plugins includes a security software plugin that is capable of preventing accesses of data included in the bitmap index are not compliant with a specified set of security policies.
17. The system of claim 12 , wherein the set of software plugins includes a monitoring software plugin that is capable of collecting a set of metrics pertaining to an operation of the particular VDS.
18. The system of claim 12 , wherein a second VDS managed by the system is associated with a second set of software plugins that is different than the set of software plugins associated with the particular VDS.