Managing parameter sets

1. A method for managing sets of parameter values, the method including:

receiving a plurality of sets of parameter values for a generic computer program, and

processing log entries associated with executions of instances of the generic computer program, each instance of the generic computer program being associated with one or more parameter values the processing including:

analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class of parameters or a second class of parameters, including classifying parameters based on whether or not the parameters affect data lineage associated with the generic computer program, the data lineage representing dependency relationships arising from execution of one or more instances of the generic computer program;

processing a log entry associated with an execution of a first instance of the generic computer program to form a particular set of parameter values, wherein the processing includes:

including, in the particular set, any values of parameters classified as members of the first class occurring in the log entry, and

excluding, from the particular set, any values of parameters classified as members of the second class occurring in the log entry; and

determining whether to add the particular set of parameter values to the plurality of sets of parameter values based on a comparison of a first identifier for the particular set of parameter values to identifiers for at least some of the sets of parameter values of the plurality of sets of parameter values.

2. The method of claim 1 wherein the comparison of the first identifier for the particular set of parameter values to the identifiers for at least some of the sets of parameter values of the plurality of sets of parameter values includes:

determining the first identifier based on the particular set of parameter values and an identifier of the generic computer program;

determining a plurality of second identifiers, one for each set of parameter values of the at least some of the sets of parameter values; and

comparing the first identifier to each second identifier of the plurality of second identifiers to determine whether the first identifier and any of the second identifiers match.

3. The method of claim 2 wherein determining whether to add the particular set of parameter values to the plurality of sets of parameter values includes determining to add the particular set of parameter values to the plurality of sets of parameter values if none of the second identifiers match the first identifier.

4. The method of claim 2 wherein determining the first identifier includes computing an identification string from the contents of the particular set of parameter values, and determining the second identifiers includes computing identification strings from the contents of the at least some of the sets of parameter values of the plurality of sets of parameter values.

5. The method of claim 2 wherein determining the first identifier includes forming a concatenation of one or more of the identifier of the generic computer program, name-value pairs of the particular set of parameter values, a function prototype of the generic computer program, and a project scope for the first instance of the generic computer program.

6. The method of claim 2 wherein determining the first identifier includes applying a data mapping function to one or more of the identifier of the generic computer program, name-value pairs of the particular set of parameter values, a function prototype of the generic computer program, and a project scope for the first instance of the generic computer program.

7. The method of claim 6 wherein the data mapping function includes a hash function.

8. The method of claim 1 wherein the first class of parameters includes parameters that affect the logical operation of the generic computer program and the second class of parameters includes parameters that do not affect the logical operation of the generic computer program.

9. The method of claim 1 wherein the generic computer program is specified as a dataflow graph that includes nodes representing data processing operations and links between the nodes representing flows of data elements between the data processing operations.

10. The method of claim 1 wherein, for each of the parameters, the analysis includes automatically classifying the parameter.

11. The method of claim 10 wherein automatically classifying the parameter includes initially classifying the parameter as belonging to the first class of parameters, determining a number of unique values of the parameter in a plurality of executions of instances of the generic computer program, and re-classifying the parameter as belonging to the second class of parameters if the number of unique values of the parameter exceeds a predetermined threshold.

12. The method of claim 10 wherein automatically classifying the parameter includes initially classifying the parameter as belonging to the first class of parameters, determining whether or not changes in values of the parameter in a plurality of executions of instances of the generic computer program affect data lineage associated with the generic computer program, and re-classifying the parameter as belonging to the second class of parameters if changes in values of the parameter do not affect the data lineage.

13. The method of claim 1 further including forming an association between the log entry associated with the execution of the first instance of the generic computer program and the particular set of parameter values.

14. The method of claim 1 wherein the log entry associated with the execution of the first instance of the generic computer program includes a log entry of an execution command used to instantiate the generic computer program including one or more parameter values supplied as arguments to the execution command.

15. The method of claim 14 wherein the log entry associated with the execution of the first instance of the generic computer program further includes one or more of an indication of the project that the first instance executed in, an indication of internal parameters for the first instance, and an indication of environmental settings, global variables, and configuration variables used by the first instance.

16. The method of claim 10 further including processing an overall plurality of sets of parameter values for a plurality of generic computer programs and an overall plurality of log entries associated with executions of instances of at least some of the plurality of generic computer programs to form a data lineage report, wherein the overall plurality of sets of parameter values includes the augmented plurality of sets of parameter values for the generic computer program, and the overall plurality of log entries associated with executions of instances of the at least some of the plurality of generic computer programs includes the log entry of the execution of the first instance of the generic computer program, including its association with the particular set of parameter values.

17. The method of claim 16 , wherein forming the data lineage report includes, for each set of parameter values of the overall plurality of sets of parameter values for the plurality of generic computer programs,

processing the overall plurality of log entries associated with executions of instances of the at least some of the plurality of generic computer programs to identify all log entries associated with executions of instances of the generic computer program corresponding to the set of parameter values, and identifying a most recent time of instantiation of the generic computer program from the identified log entries associated with executions of instances of the generic computer program; and

determining whether to include the set of parameters in the data lineage report based on the most recent time of instantiation of the generic computer program.

18. The method of claim 17 wherein determining whether to include the set of parameters in the data lineage report based on the most recent time of instantiation of the generic computer program includes comparing the most recent time of instantiation to a predetermined time interval and including the set of parameters in the data lineage report if the most recent time of instantiation of the generic computer program is within the predetermined time interval.

19. The method of claim 16 wherein forming the data lineage report includes, for each set of parameter values of the overall plurality of sets of parameter values for the plurality of generic computer programs,

processing the overall plurality of log entries associated with executions of instances of the at least some of the plurality of generic computer programs to determine a number of log entries associated with executions of instances of the generic computer program corresponding to the set of parameter values, and

determining whether to include the set of parameters in the data lineage report based on the number of log entries associated with executions of instances of the generic computer program.

20. Software stored in a non-transitory form on a computer-readable medium, for managing sets of parameter values, the software including instructions for causing a computing system to:

receive a plurality of sets of parameter values for a generic computer program, and

process log entries associated with executions of instances of the generic computer program, each instance of the generic computer program being associated with one or more parameter values and augmenting the plurality of sets of parameter values based on the processing, the processing including:

analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class of parameters or a second class of parameters, including classifying parameters based on whether or not the parameters affect data lineage associated with the generic computer program, the data lineage representing dependency relationships arising from execution of one or more instances of the generic computer program;

processing a log entry associated with an execution of a first instance of the generic computer program to form a particular set of parameter values, wherein the processing includes:

including, in the particular set, any values of parameters classified as members of the first class occurring in the log entry , and

excluding, from the particular set, any values of parameters classified as members of the second class occurring in the log entry; and

determining whether to add the particular set of parameter values to the plurality of sets of parameter values based on a comparison of a first identifier for the particular set of parameter values to identifiers for at least some of the sets of parameter values of the plurality of sets of parameter values.

21. A computing system for managing sets of parameter values, the computing system including:

an input device or port for receiving a plurality of sets of parameter values for a generic computer program,

a memory storing instructions, and

at least one processor for processing according to the instructions log entries associated with executions of instances of the generic computer program, each instance of the generic computer program being associated with one or more parameter, the processing including:

analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class of parameters or a second class of parameters, including classifying parameters based on whether or not the parameters affect data lineage associated with the generic computer program, the data lineage representing dependency relationships arising from execution of one or more instances of the generic computer program;

processing a log entry associated with an execution of a first instance of the generic computer program to form a particular set of parameter values, wherein the processing includes:

including, in the particular set, any values of parameters classified as members of the first class occurring in the log entry, and

excluding, from the particular set, any values of parameters classified as members of the second class occurring in the log entry; and

determining whether to add the particular set of parameter values to the plurality of sets of parameter values based on a comparison of a first identifier for the particular set of parameter values to identifiers for at least some of the sets of parameter values of the plurality of sets of parameter values.

22. A computing system for managing sets of parameter values, the computing system including:

means for receiving a plurality of sets of parameter values for a generic computer program, and

means for processing log entries associated with executions of instances of the generic computer program, each instance of the generic computer program being associated with one or more parameter values the processing including:

analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class of parameters or a second class of parameters, including classifying parameters based on whether or not the parameters affect data lineage associated with the generic computer program, the data lineage representing dependency relationships arising from execution of one or more instances of the generic computer program;

processing a log entry associated with an execution of a first instance of the generic computer program to form a particular set of parameter values, wherein the processing includes:

including, in the particular set, any values of parameters classified as members of the first class occurring in the log entry, and

excluding, from the particular set, any values of parameters classified as members of the second class occurring in the log entry; and

determining whether to add the particular set of parameter values to the plurality of sets of parameter values based on a comparison of a first identifier for the particular set of parameter values to identifiers for at least some of the sets of parameter values of the plurality of sets of parameter values.

23. The method of claim 1 wherein each of the log entries associated with executions of instances of the generic computer program was previously stored in a data store.

24. The software of claim 20 wherein the comparison of the first identifier for the particular set of parameter values to the identifiers for at least some of the sets of parameter values of the plurality of sets of parameter values includes:

determining the first identifier based on the particular set of parameter values and an identifier of the generic computer program;

determining a plurality of second identifiers, one for each set of parameter values of the at least some of the sets of parameter values; and

comparing the first identifier to each second identifier of the plurality of second identifiers to determine whether the first identifier and any of the second identifiers match.

25. The software of claim 24 wherein determining whether to add the particular set of parameter values to the plurality of sets of parameter values includes determining to add the particular set of parameter values to the plurality of sets of parameter values if none of the second identifiers match the first identifier.

26. The software of claim 24 wherein determining the first identifier includes computing an identification string from the contents of the particular set of parameter values, and determining the second identifiers includes computing identification strings from the contents of the at least some of the sets of parameter values of the plurality of sets of parameter values.

27. The software of claim 24 wherein determining the first identifier includes forming a concatenation of one or more of the identifier of the generic computer program, name-value pairs of the particular set of parameter values, a function prototype of the generic computer program, and a project scope for the first instance of the generic computer program.

28. The software of claim 24 wherein determining the first identifier includes applying a data mapping function to one or more of the identifier of the generic computer program, name-value pairs of the particular set of parameter values, a function prototype of the generic computer program, and a project scope for the first instance of the generic computer program.

29. The software of claim 28 wherein the data mapping function includes a hash function.

30. The software of claim 20 wherein the first class of parameters includes parameters that affect the logical operation of the generic computer program and the second class of parameters includes parameters that do not affect the logical operation of the generic computer program.

31. The software of claim 20 wherein the generic computer program is specified as a dataflow graph that includes nodes representing data processing operations and links between the nodes representing flows of data elements between the data processing operations.

32. The software of claim 20 wherein, for each of the parameters, the analysis includes automatically classifying the parameter.

33. The software of claim 32 wherein automatically classifying the parameter includes initially classifying the parameter as belonging to the first class of parameters, determining a number of unique values of the parameter in a plurality of executions of instances of the generic computer program, and re-classifying the parameter as belonging to the second class of parameters if the number of unique values of the parameter exceeds a predetermined threshold.

34. The software of claim 32 wherein automatically classifying the parameter includes initially classifying the parameter as belonging to the first class of parameters, determining whether or not changes in values of the parameter in a plurality of executions of instances of the generic computer program affect data lineage associated with the generic computer program, and re-classifying the parameter as belonging to the second class of parameters if changes in values of the parameter do not affect the data lineage.

35. The software of claim 20 further including forming an association between the log entry associated with the execution of the first instance of the generic computer program and the particular set of parameter values.

36. The software of claim 20 wherein the log entry associated with the execution of the first instance of the generic computer program includes a log entry of an execution command used to instantiate the generic computer program including one or more parameter values supplied as arguments to the execution command.

37. The software of claim 36 wherein the log entry associated with the execution of the first instance of the generic computer program further includes one or more of an indication of the project that the first instance executed in, an indication of internal parameters for the first instance, and an indication of environmental settings, global variables, and configuration variables used by the first instance.

38. The software of claim 32 wherein the processing further includes processing an overall plurality of sets of parameter values for a plurality of generic computer programs and an overall plurality of log entries associated with executions of instances of at least some of the plurality of generic computer programs to form a data lineage report, wherein the overall plurality of sets of parameter values includes the augmented plurality of sets of parameter values for the generic computer program, and the overall plurality of log entries associated with executions of instances of the at least some of the plurality of generic computer programs includes the log entry of the execution of the first instance of the generic computer program, including its association with the particular set of parameter values.

39. The software of claim 38 , wherein forming the data lineage report includes, for each set of parameter values of the overall plurality of sets of parameter values for the plurality of generic computer programs,

processing the overall plurality of log entries associated with executions of instances of the at least some of the plurality of generic computer programs to identify all log entries associated with executions of instances of the generic computer program corresponding to the set of parameter values, and identifying a most recent time of instantiation of the generic computer program from the identified log entries associated with executions of instances of the generic computer program; and

determining whether to include the set of parameters in the data lineage report based on the most recent time of instantiation of the generic computer program.

40. The software of claim 39 wherein determining whether to include the set of parameters in the data lineage report based on the most recent time of instantiation of the generic computer program includes comparing the most recent time of instantiation to a predetermined time interval and including the set of parameters in the data lineage report if the most recent time of instantiation of the generic computer program is within the predetermined time interval.

41. The software of claim 38 wherein forming the data lineage report includes, for each set of parameter values of the overall plurality of sets of parameter values for the plurality of generic computer programs,

processing the overall plurality of log entries associated with executions of instances of the at least some of the plurality of generic computer programs to determine a number of log entries associated with executions of instances of the generic computer program corresponding to the set of parameter values, and

determining whether to include the set of parameters in the data lineage report based on the number of log entries associated with executions of instances of the generic computer program.

42. The method of claim 1 wherein, for each of the parameters, the analysis includes accepting a user-defined classification for the parameter.

43. The software of claim 20 wherein, for each of the parameters, the analysis includes accepting a user-defined classification for the parameter.