INTELLIGENT DATA INTEGRATION SYSTEM
A computing system receives an indication of one or more data sources for a data integration process. The computing system interfaces with the one or more data sources to receive the data for analysis. The computing system analyzes using one or more artificial intelligence models or one or more statistical models to recognize patterns and relationships within the data by identifying commonalities, dependencies, and structures that exist across the data. The computing system generates a mapping strategy based on the identified patterns and the identified relationships in the data. The computing system maps the data from the one or more data sources to the selected destination in accordance with the mapping strategy. The computing system generates a graphical user interface that visualizes the mapping strategy for display on a user device.
1 .- 20 . (canceled)
21 . A method, comprising:
receiving, by a computing system via a graphical user interface, a selection of a data transformation to be applied during a data mapping;
generating, by the computing system, a preview of the selected data transformation by processing a sample of dataset records according to parameters for the selected data transformation;
rendering, by the computing system, the preview within an overlay window in the graphical user interface;
receiving, by the computing system via the overlay window, one or more user-specified transformation parameters;
updating, by the computing system, the preview in real time or near real-time by reprocessing the sample according to the one or more user-specified transformation parameters; and
responsive to receiving a user confirmation, applying, by the computing system, the selected data transformation with the one or more user-specified transformation parameters to the dataset record during execution of the data mapping.
22 . The method of claim 21 , wherein generating, by the computing system, the preview of the selected data transformation comprises:
sampling the dataset records using one of a random sampling technique or a criteria-based sampling technique selected to ensure responsiveness for large datasets.
23 . The method of claim 21 , further comprising:
receiving, by the computing system, adjustments in real-time or near real-time, the adjustments affecting the preview shown in the overlay window.
24 . The method of claim 21 , wherein the overlay window includes action controls comprising one or more of:
a preview button to trigger generation of the preview on the sample of the dataset record;
an apply button to commit the selected data transformation with the one or more user-specified parameters;
a pause button to pause processing of the preview;
a duplicate button to replicate current settings for further modification; or
a remove button to remove the selected data transformation.
25 . The method of claim 21 , wherein updating, by the computing system, the preview in real time or near real-time by reprocessing the sample of the dataset record according to the one or more user-specified transformation parameters occurs while the computing system continues to ingest streaming data from a real-time source, and the selected data transformation is applied to subsequent streaming data records.
26 . The method of claim 21 , further comprising:
prior to applying the selected data transformation to the dataset records, validating, by the computing system, the one or more user-specified transformation parameters for consistency and presenting an in-interface notification if a parameter is inconsistent with a data type or a missing dependency.
27 . The method of claim 21 , wherein the preview is presented as at least one of a tabular table or a chart that visually represents effects of the selected data transformation on the sampled dataset.
28 . A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by a processor, causes a computing system to perform operations comprising:
receiving, by the computing system via a graphical user interface, a selection of a data transformation to be applied during a data mapping;
generating, by the computing system, a preview of the selected data transformation by processing a sample of dataset records according to parameters for the selected data transformation;
rendering, by the computing system, the preview within an overlay window in the graphical user interface;
receiving, by the computing system via the overlay window, one or more user-specified transformation parameters;
updating, by the computing system, the preview in real time or near real-time by reprocessing the sample according to the one or more user-specified transformation parameters; and
responsive to receiving a user confirmation, applying, by the computing system, the selected data transformation with the one or more user-specified transformation parameters to the dataset record during execution of the data mapping.
29 . The non-transitory computer readable medium of claim 28 , wherein generating, by the computing system, the preview of the selected data transformation comprises:
sampling the dataset records using one of a random sampling technique or a criteria-based sampling technique selected to ensure responsiveness for large datasets.
30 . The non-transitory computer readable medium of claim 28 , further comprising:
receiving, by the computing system, adjustments in real-time or near real-time, the adjustments affecting the preview shown in the overlay window.
31 . The non-transitory computer readable medium of claim 28 , wherein the overlay window includes action controls comprising one or more of:
a preview button to trigger generation of the preview on the sample of the dataset record;
an apply button to commit the selected data transformation with the one or more user-specified parameters;
a pause button to pause processing of the preview;
a duplicate button to replicate current settings for further modification; or
a remove button to remove the selected data transformation.
32 . The non-transitory computer readable medium of claim 28 , wherein updating, by the computing system, the preview in real time or near real-time by reprocessing the sample of the dataset record according to the one or more user-specified transformation parameters occurs while the computing system continues to ingest streaming data from a real-time source, and the selected data transformation is applied to subsequent streaming data records.
33 . The non-transitory computer readable medium of claim 28 , further comprising:
prior to applying the selected data transformation to the dataset records, validating, by the computing system, the one or more user-specified transformation parameters for consistency and presenting an in-interface notification if a parameter is inconsistent with a data type or a missing dependency.
34 . The non-transitory computer readable medium of claim 28 , wherein the preview is presented as at least one of a tabular table or a chart that visually represents effects of the selected data transformation on the sampled dataset.
35 . A system, comprising:
a processor; and
a memory having computing instructions stored thereon, which, when executed by the processor, causes the system to perform operations comprising:
receiving, via a graphical user interface, a selection of a data transformation to be applied during a data mapping;
generating a preview of the selected data transformation by processing a sample of dataset records according to parameters for the selected data transformation;
rendering the preview within an overlay window in the graphical user interface;
receiving, via the overlay window, one or more user-specified transformation parameters;
updating the preview in real time or near real-time by reprocessing the sample according to the one or more user-specified transformation parameters; and
responsive to receiving a user confirmation, applying the selected data transformation with the one or more user-specified transformation parameters to the dataset record during execution of the data mapping.
36 . The system of claim 35 , wherein generating the preview of the selected data transformation comprises:
sampling the dataset records using one of a random sampling technique or a criteria-based sampling technique selected to ensure responsiveness for large datasets.
37 . The system of claim 35 , wherein the operations further comprise:
receiving adjustments in real-time or near real-time, the adjustments affecting the preview shown in the overlay window.
38 . The system of claim 35 , wherein the overlay window includes action controls comprising one or more of:
a preview button to trigger generation of the preview on the sample of the dataset record;
an apply button to commit the selected data transformation with the one or more user-specified parameters;
a pause button to pause processing of the preview;
a duplicate button to replicate current settings for further modification; or
a remove button to remove the selected data transformation.
39 . The system of claim 35 , wherein updating the preview in real time or near real-time by reprocessing the sample of the dataset record according to the one or more user-specified transformation parameters occurs while the computing system continues to ingest streaming data from a real-time source, and the selected data transformation is applied to subsequent streaming data records.
40 . The system of claim 35 , wherein the operations further comprise:
prior to applying the selected data transformation to the dataset records, validating the one or more user-specified transformation parameters for consistency and presenting an in-interface notification if a parameter is inconsistent with a data type or a missing dependency.