System and methods for web data transformation sourcing
A computer-implemented system for web data transformation sourcing is disclosed to include a search module defined to receive a set of original input data types and a set of ultimate output data types. The search module is defined to locate one or more web based sources defined to transform the set of original input data types into the set of ultimate output data types. The search module is further defined to generate a transformation solution that when executed utilizes the one or more located web based sources to transform the set of original input data types into the set of ultimate output data types. The transformation solution is digitally conveyed.
1. A computer-implemented system for web data transformation sourcing, comprising:
a search module defined to locate one or more web based sources defined to transform a set of original input data types into a set of ultimate output data types, wherein each original input data type in the set of original input data types is a respective structured piece of data that has semantic meaning, and wherein each ultimate output data type in the set of ultimate output data types is a respective structured piece of data that has semantic meaning, the search module defined to determine multiple transformation paths by which the set of original input data types is transformed into the set of ultimate output data types, the search module further defined to generate and digitally convey a transformation solution that when executed utilizes the one or more located web based sources to transform the set of original input data types into the set of ultimate output data types, wherein the search module is defined to generate a cost value for each of the multiple transformation paths, wherein each cost value is generated in accordance with a cost function.
2. The computer-implemented system for web data transformation sourcing as recited in claim 1 , wherein the set of original input data types and the set of ultimate output data types correspond to data types known to the one or more located web based sources.
3. The computer-implemented system for web data transformation sourcing as recited in claim 1 , wherein the one or more web based sources correspond to respective application programming interfaces (API's), each API defined to operate on input data corresponding to one or more input data types to generate output data corresponding to one or more output data types associated with the one or more input data types.
4. The computer-implemented system for web data transformation sourcing as recited in claim 1 , wherein the transformation solution is defined as a universal resource locator (URL) that references and links the one or more located web based sources.
5. The computer-implemented system for web data transformation sourcing as recited in claim 1 , wherein the search module is further defined to determine that one or more web based sources are not available to transform the set of original input data types into the set of ultimate output data types.
6. The computer-implemented system for web data transformation sourcing as recited in claim 5 , wherein the search module is defined to determine and digitally convey one or more alternate output data types into which the set of original input data types can be transformed by the available web based sources.
7. The computer-implemented system for web data transformation sourcing as recited in claim 5 , wherein the search module is defined to determine and digitally convey a number of additional input data types that when combined with the set of original input data types will provide for location of one or more web based sources defined to transform the set of original input data types and the number of additional input data types into the set of ultimate output data types.
8. The computer-implemented system for web data transformation sourcing as recited in claim 1 , wherein the search module is defined to combine multiple web based sources in the transformation solution by automatically supplying the output of some web based sources to the input of other web based sources until the set of ultimate output data types is obtained.
9. The computer-implemented system for web data transformation sourcing as recited in claim 1 , wherein the search module is defined to generate an index of web based sources as the search module operates to locate one or more web based sources defined to transform the set of original input data types into the set of ultimate output data types, the index of web based sources including an identifier of each examined web based source and its corresponding input and output data types.
10. The computer-implemented system for web data transformation sourcing as recited in claim 9 , wherein the search module is defined to search the index of web based sources prior to performing a web search to locate one or more web based sources defined to transform the set of original input data types into the set of ultimate output data types.
11. The computer-implemented system for web data transformation sourcing as recited in claim 1 , wherein the cost function accounts for latency and restrictions associated with each of one or more segments which define a given transformation path.
12. The computer-implemented system for web data transformation sourcing as recited in claim 1 , further comprising:
an application programming interface (API) generation module defined to combine multiple API's into single API to perform a particular data type transformation, wherein the multiple API's are located by the search module when operated to locate one or more web based sources defined to transform the set of original input data types into the set of ultimate output data types.
13. The computer-implemented system for web data transformation sourcing as recited in claim 1 , further comprising:
an interface module defined to receive the set of original input data types and the set of ultimate output data types, wherein an interfacing portion of the interface module is implemented as either a graphical user interface, a web service, or a command line.
14. A method for web data transformation sourcing, comprising:
providing a set of original input data types as input to a web source search engine, wherein each original input data type in the set of original input data types is a respective structured piece of data that has semantic meaning;
providing a set of ultimate output data types as input to the web source search engine, wherein each ultimate output data type in the set of ultimate output data types is a respective structured piece of data that has semantic meaning;
operating the web source search engine to automatically search for one or more web based sources that will transform the set of original input data types into the set of ultimate output data types and to determine multiple transformation paths by which the set of original input data types is transformed into the set of ultimate output data types;
upon locating the one or more web based sources, operating the web source search engine to automatically generate a universal resource locator (URL) that upon execution will utilize the one or more web based sources to provide output data corresponding to the set of ultimate output data types based on input data corresponding to the set of original input data types;
digitally conveying the generated URL for use in web data transformation in which output data corresponding to the set of ultimate output data types is to be generated based on input data corresponding to the set of original input data types; and
operating the web source search engine to generate a cost value for each of the multiple transformation paths, wherein each cost value is generated in accordance with a cost function.
15. The method for web data transformation sourcing as recited in claim 14 , wherein the set of original input data types and the set of ultimate output data types are respectively obtained from a data type hierarchy, wherein the data type hierarchy includes a description of each data type known within an existing population of web based sources.
16. The method for web data transformation sourcing as recited in claim 14 , further comprising:
upon failing to locate one or more web based sources that will transform the set of original input data types into the set of ultimate output data types, operating the web source search engine to automatically determine an expanded set of original input data types that will provide for location of one or more web based sources defined to transform the expanded set of original input data types into the set of ultimate output data types;
digitally conveying the expanded set of original input data types; and
digitally conveying the generated URL for use in web data transformation in which output data corresponding to the set of ultimate output data types is to be generated based on input data corresponding to the expanded set of original input data types.
17. The method for web data transformation sourcing as recited in claim 14 , further comprising:
upon failing to locate one or more web based sources that will transform the set of original input data types into the set of ultimate output data types, operating the web source search engine to automatically determine one or more alternate output data types into which the set of original input data types can be transformed by the available web based sources;
digitally conveying the one or more alternate output data types; and
digitally conveying the generated URL for use in web data transformation in which output data corresponding to the one or more alternate output data types is to be generated based on input data corresponding to the set of original input data types.
18. The method for web data transformation sourcing as recited in claim 14 , wherein the cost function accounts for latency and restrictions associated with each of one or more segments which define a given transformation path.
19. The method for web data transformation sourcing as recited in claim 14 , further comprising:
generating an index of web based sources as the web source search engine operates to automatically search for one or more web based sources that will transform the set of original input data types into the set of ultimate output data types, wherein the index of web based sources includes an identifier of each examined web based source and its corresponding input and output data types.
20. The method for web data transformation sourcing as recited in claim 19 , further comprising:
operating the web source search engine to search the index of web based sources for one or more web based sources that will transform the set of original input data types into the set of ultimate output data types.
21. The method for web data transformation sourcing as recited in claim 14 , further comprising:
operating the web source search engine to search open data tables that include references to web based sources and associated input and output data types as part of searching for one or more web based sources that will transform the set of original input data types into the set of ultimate output data types.
22. The method for web data transformation sourcing as recited in claim 21 , wherein each open data table is accessible through an associated URL, and wherein the open data table includes binding data which binds one or more web data sources referenced within the open data table to a query language web service, the binding data including instructions to the query language web service with regard to querying the one or more web data sources for specific data present at the one or more web data sources.
23. The method for web data transformation sourcing as recited in claim 22 , wherein the query language web service is defined to expose a query language for specification of a particular web data source to be queried, data types to be queried within the particular web data source, and one or more operations to be performed on the particular web data source.
24. The method for web data transformation sourcing as recited in claim 23 , wherein the generated URL is defined to be executable through the query language web service.
25. A method for web data transformation sourcing, comprising:
providing a set of original input data types as input to a web source search engine, wherein each original input data type in the set of original input data types is a respective structured piece of data that has semantic meaning;
operating the web source search engine to automatically search for one or more web based sources that will transform the set of original input data types into one or more output data types, wherein each of the one or more output data types is a respective structured piece of data that has semantic meaning;
upon locating the one or more web based sources, operating the web source search engine to automatically generate a plurality of universal resource locators (URLs) as a corresponding plurality of transformation solutions that each upon execution will utilize at least one of the one or more web based sources to provide output data corresponding to the one or more output data types based on input data corresponding to the set of original input data types;
determining a cost value of each of the plurality of transformation solutions, wherein each cost value is determined in accordance with a cost function; and
digitally conveying the one or more output data types and the plurality of transformation solutions and the cost values of the plurality of transformation solutions.
26. A method for web data transformation sourcing, comprising:
providing a set of ultimate output data types as input to a web source search engine, wherein each ultimate output data type in the set of ultimate output data types is a respective structured piece of data that has semantic meaning;
operating the web source search engine to automatically search for one or more web based sources that will transform one or more input data types into the set of ultimate output data types, wherein each of the one or more input data types is a respective structured piece of data that has semantic meaning;
upon locating the one or more web based sources, operating the web source search engine to automatically generate a plurality of universal resource locators (URLs) as a corresponding plurality of transformation solutions that each upon execution will utilize at least one of the one or more web based sources to provide output data corresponding to the set of ultimate output data types based on input data corresponding to the one or more input data types;
determining a cost value of each of the plurality of transformation solutions, wherein each cost value is determined in accordance with a cost function; and
digitally conveying the one or more input data types and the plurality of transformation solutions and the cost values of the plurality of transformation solutions.