Managing metadata for graph-based computations
Determining metadata associated with a graph-based computation includes functionally transforming metadata associated with a first portion of a graph to generate transformed metadata associated with a second portion of the graph; determining a third portion of the graph related to the second portion of the graph; and propagating the transformed metadata from the second portion of the graph to the third portion of the graph.
1. A method performed by one or more processing devices, the method including:
receiving a graph of components and links, with at least one of the components representing one or more processes implemented by a system in processing data, each of the components having zero or more ports, each link between at least two ports, with at least one of the components being associated with information for receiving a data set that is stored in hardware or being associated with information for storing a data set in hardware;
generating a list that specifies an order of at least two of the components;
identifying a first component from the list;
receiving a specification associated with the first component;
identifying a first record format associated with a first port of the first component, with the first port representing receipt of data by the first component;
generating a second record format associated with a second port of the first component by transforming the first record format according to the specification associated with the first component, with the second port representing output of data from the first component;
identifying a second component linked to the first component by a link connecting the second port of the first component to a particular port of the second component;
associating the second record format with the particular port; and
in response to associating the second record format with the particular port of the second component, placing information representing the second component on the list.
2. The method of claim 1 , wherein generating the list includes ordering components according to a partial ordering determined by the links.
3. The method of claim 1 , further including:
receiving a request from a device of a user; and
causing a display of a record format associated with a port to the user in response to the request.
4. The method of claim 3 , wherein the request from the user includes input from the user selecting the port for which record format is to be displayed.
5. The method of claim 4 , wherein the input from the user includes positioning an on-screen pointer near a graphical representation of port for a predetermined amount of time.
6. The method of claim 5 , wherein the record format is displayed before the graph is run.
7. A non-transitory computer storage medium encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving a graph of components and links, with at least one of the components representing one or more processes implemented by a system in processing data, each of the components having zero or more ports, each link between at least two ports, with at least one of the components being associated with information for receiving a data set that is stored in hardware or being associated with information for storing a data set in hardware;
generating a list that specifies an order of at least two of the components;
identifying a first component from the list;
receiving a specification associated with the first component;
identifying a first record format associated with a first port of the first component, with the first port representing receipt of data by the first component;
generating a second record format associated with a second port of the first component by transforming the first record format according to the specification associated with the first component, with the second port representing output of data from the first component;
identifying a second component linked to the first component by a link connecting the second port of the first component to a particular port of the second component;
associating the second record format with the particular port; and
in response to associating the second record format with the particular port of the second component, placing information representing the second component on the list.
8. The medium of claim 7 , wherein generating the list includes ordering components according to a partial ordering determined by the links.
9. The medium of claim 7 , wherein the operations further include:
receiving a request from a device of a user; and
causing a display of a record format associated with a port to the user in response to the request.
10. The medium of claim 9 , wherein the request from the user includes input from the user selecting the port for which record format is to be displayed.
11. The medium of claim 10 , wherein the input from the user includes positioning an on-screen pointer near a graphical representation of port for a predetermined amount of time.
12. The medium of claim 11 , wherein the record format is displayed before the graph is run.
13. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving a graph of components and links, with at least one of the components representing one or more processes implemented by a system in processing data, each of the components having zero or more ports, each link between at least two ports, with at least one of the components being associated with information for receiving a data set that is stored in hardware or being associated with information for storing a data set in hardware;
generating a list that specifies an order of at least two of the components;
identifying a first component from the list;
receiving a specification associated with the first component;
identifying a first record format associated with a first port of the first component, with the first port representing receipt of data by the first component;
generating a second record format associated with a second port of the first component by transforming the first record format according to the specification associated with the first component, with the second port representing output of data from the first component;
identifying a second component linked to the first component by a link connecting the second port of the first component to a particular port of the second component;
associating the second record format with the particular port; and
in response to associating the second record format with the particular port of the second component, placing information representing the second component on the list.
14. The system of claim 13 , wherein generating the list includes ordering components according to a partial ordering determined by the links.
15. The system of claim 13 , wherein the operations further include:
receiving a request from a device of a user; and
causing a display of a record format associated with a port to the user in response to the request.
16. The system of claim 15 , wherein the request from the user includes input from the user selecting the port for which record format is to be displayed.
17. The system of claim 16 , wherein the input from the user includes positioning an on-screen pointer near a graphical representation of port for a predetermined amount of time.
18. The system of claim 17 , wherein the record format is displayed before the graph is run.