Universal adapter for vendor data
An electronic online system is configured to receive, at the electronic online system, an expression of a use case; determine, using a first machine-learning technique with the expression of the use case as input, a data source to satisfy the use case; determine, using a second machine-learning technique with the expression of the use case and the inference of the first machine-learning technique as inputs, a data destination to satisfy the use case; and construct a data pipeline from the data source to the data destination for the use case.
1. An electronic system comprising:
a processor subsystem; and
a memory including instructions, which when executed by the processor subsystem, cause the processor subsystem to:
receive, at the electronic system, an unstructured expression of a use case;
automatically determine, using a first trained machine-learning technique with the unstructured expression of the use case as input, a vendor-specific format and a data source to satisfy the use case;
automatically determine, using a second trained machine-learning technique with both (i) the unstructured expression of the use case and (ii) an inference output from the first trained machine-learning technique as inputs, an organization-specific data destination format and a data destination to satisfy the use case;
construct a real-time data pipeline from the data source to the data destination for the use case by:
configuring an ingest connector to obtain vendor data in the vendor-specific format from the data source,
transforming the vendor data from the vendor-specific format to the organization-specific data destination format using the inference output, to obtain transformed data, and
configuring an export connector to transmit the transformed data to the data destination; and
automatically update the data destination in real-time with vendor data through the real-time data pipeline using a publication-subscription model.
2. The electronic system of claim 1 , wherein the unstructured expression of the use case is formed as a query.
3. The electronic system of claim 1 , wherein the unstructured expression of the use case is formed as a business objective.
4. The electronic system of claim 1 , wherein the unstructured expression of the use case is formed as a description of an output.
5. The electronic system of claim 1 , wherein the unstructured expression of the use case does not include the data source.
6. The electronic system of claim 1 , wherein the data source includes a database with a SQL database structure.
7. The electronic system of claim 1 , wherein the data source includes a database with a NoSQL database structure.
8. The electronic system of claim 1 , wherein the data destination includes a database with a SQL database structure.
9. The electronic system of claim 1 , wherein the data destination includes a database with a NoSQL database structure.
10. The electronic system of claim 1 , wherein the real-time data pipeline includes an ingest Kafka Connector to obtain data from the data source, an export Kafka Connector to transmit data to the data destination, and a Kafka topic to store a configuration of the ingest Kafka Connector and the export Kafka Connector.
11. The electronic system of claim 1 , wherein the real-time data pipeline is a publication-subscription information sharing model with the data destination being a subscriber.
12. A method performed on an electronic online system, the method comprising:
receiving, at the electronic online system, an unstructured expression of a use case;
automatically determining, using a first trained machine-learning technique with the unstructured expression of the use case as input, a vendor-specific format and a data source to satisfy the use case;
automatically determining, using a second trained machine-learning technique with both (i) the unstructured expression of the use case and (ii) an inference output from the first trained machine-learning technique as inputs, an organization-specific data destination format and data destination to satisfy the use case;
constructing a real-time data pipeline from the data source to the data destination for the use case by:
configuring an ingest connector to obtain vendor data in the vendor-specific format from the data source,
transforming the vendor data from the vendor-specific format to the organization-specific data destination format using the inference output, to obtain transformed data, and
configuring an export connector to transmit the transformed data to the data destination; and
automatically updating the data destination in real-time with vendor data through the real-time data pipeline using a publication-subscription model.
13. The method of claim 12 , wherein the unstructured expression of the use case is formed as a query.
14. The method of claim 12 , wherein the unstructured expression of the use case is formed as a business objective.
15. The method of claim 12 , wherein the unstructured expression of the use case is formed as a description of an output.
16. The method of claim 12 , wherein the unstructured expression of the use case does not include the data source.
17. The method of claim 12 , wherein the real-time data pipeline includes an ingest Kafka Connector to obtain data from the data source, an export Kafka Connector to transmit data to the data destination, and a Kafka topic to store a configuration of the ingest Kafka Connector and the export Kafka Connector.
18. The method of claim 12 , wherein the real-time data pipeline is a publication-subscription information sharing model with the data destination being a subscriber.
19. A non-transitory machine-readable medium comprising instructions, which when executed by a machine in an electronic online system, cause the machine to:
receive, at the electronic online system, an unstructured expression of a use case;
automatically determine, using a first trained machine-learning technique with the unstructured expression of the use case as input, a vendor-specific format and a data source to satisfy the use case;
automatically determine, using a second trained machine-learning technique with both (i) the unstructured expression of the use case and (ii) an inference output from the first trained machine-learning technique as inputs, an organization-specific data destination format and a data destination to satisfy the use case;
construct a real-time data pipeline from the data source to the data destination for the use case by:
configuring an ingest connector to obtain vendor data in the vendor-specific format from the data source,
transforming the vendor data from the vendor-specific format to the organization-specific data destination format using the inference output, to obtain transformed data, and
configuring an export connector to transmit the transformed data to the data destination; and
automatically update the data destination in real-time with vendor data through the real-time data pipeline using a publication-subscription model.
20. The non-transitory machine-readable medium of claim 19 , wherein the real-time data pipeline is a publication-subscription information sharing model with the data destination being a subscriber.