Method and apparatus for enhancing reliability of automated data processing
View Patent ↗A number of items of data from a data source ( 12 ) can be processed and supplied to a data destination ( 16, 17 ). The data may include image data, text data, numeric data or other types of data, or combination of these types of data. The processing of the data is controlled by a project definition ( 14, 71, 101 ), which includes a plurality of modules selected from a variety of available modules (Tables 1-4). The modules have input and output ports which are interrelated by binding information. Capability is provided for automated detection and resolution of certain types of problems that may arise during automated data processing, including progressively inefficient use of memory, as well as development of an abnormal processing situation in which no meaningful data processing is taking place.
1. A method comprising the steps of:
executing in a computer system a first procedure which selectively launches execution in said computer system of a predetermined project definition of a set of predetermined project definitions in response to respective requests for execution thereof,
wherein each said predetermined project definition is for manipulating data and comprises:
a plurality of function portions which each correspond to one of a plurality of predetermined function definitions that are different, each predetermined function definition defining a process to be applied to the data and at least one input port and at least one output port that are functionally related according to the corresponding function definition;
a further portion which includes a source portion identifying a data source and defining an output port through which data from the data source can be produced, and which includes a destination portion identifying a data destination and defining an input port through which data can be supplied to the data destination;
binding information which includes binding portions that each associate a respective said input port with one of said output ports; and
executing in said computer system a second procedure in association with at least one of the first procedure and the predetermined project definition,
wherein the second procedure, when executed, repeatedly performs a series of steps, comprising:
monitoring a predetermined performance characteristic of a set of predetermined performance characteristics;
determining an action relating to the at least one of the first procedure and the predetermined project definition in dependence upon at least the predetermined performance characteristic,
wherein the action relates to at least one of the first procedure and the predetermined definition, and
wherein the action comprises at least one of:
terminating the at least one of the first procedure and the predetermined project definition,
pausing execution of the at least one of the first procedure and the predetermined project definition without awaiting completion of the at least one of the first procedure and the predetermined project definition,
pausing execution of the at least one of the first procedure and the predetermined project definition upon completing execution of the current at least one of the first procedure and the predetermined project definition, and
re-starting execution of the current at least one of the first procedure and the predetermined project definition; and
causing execution of the determined action;
queuing input information destined for the first procedure upon execution of the determined action; and
submitting said queued input information to said first procedure upon re-starting execution of the at least one of the first procedure and the predetermined project definition.
2. A method according to claim 1 wherein the computer system comprises at least first and second processors, the first procedure being executed by the first processor and the second procedure being executed by the second processor.
3. A method according to claim 1 wherein,
the predetermined performance characteristic relates to at least one of memory utilization of a processor within the computer system, utilization of a processor within the computer system, status of a processor process, utilization of a network forming part of the computer system, and communications activity of a network forming part of the computer system.
4. A method according to claim 1 wherein,
re-starting execution of at least one of the first procedure and the predetermined project definition is carried out so as to resume execution of one of the at least one of the first procedure and the predetermined project definition which was interrupted, beginning with data from the associated data source which immediately follows the last data therefrom which was processed to completion and stored in the associated data destination.
5. A method according to claim 1 wherein,
the first procedure executes the set of predetermined project definitions in a predetermined order upon a set of data files stored within a memory connected to a computer network to which the computer system is also connected.
6. One or more non-transitory computer-readable storage media storing a computer program which, when executed by at least one processor, perform a method comprising:
executing in a computer system a first procedure which selectively launches execution in said computer system of a predetermined project definition of a set of predetermined project definitions in response to respective requests for execution thereof,
wherein each said predetermined project definition is for manipulating data and comprises:
a plurality of function portions which each correspond to one of a plurality of predetermined function definitions that are different, each predetermined function definition defining a process to be applied to the data and at least one input port and at least one output port that are functionally related according to the corresponding function definition;
a further portion which includes a source portion identifying a data source and defining an output port through which data from the data source can be produced, and which includes a destination portion identifying a data destination and defining an input port through which data can be supplied to the data destination;
binding information which includes binding portions that each associate a respective said input port with one of said output ports; and
executing in said computer system a second procedure in association with at least one of the first procedure and the predetermined project definition,
wherein the second procedure, when executed, repeatedly performs a series of steps, comprising:
monitoring a predetermined performance characteristic of a set of predetermined performance characteristics;
determining an action relating to the at least one of the first procedure and the predetermined project definition in dependence upon at least the predetermined performance characteristic,
wherein the action relates to at least one of the first procedure and the predetermined definition, and
wherein the action comprises at least one of:
terminating the at least one of the first procedure and the predetermined project definition,
pausing execution of the at least one of the first procedure and the predetermined project definition without awaiting completion of the at least one of the first procedure and the predetermined project definition,
pausing execution of the at least one of the first procedure and the predetermined project definition upon completing execution of the current at least one of the first procedure and the predetermined project definition, and
re-starting execution of the current at least one of the first procedure and the predetermined project definition; and
causing execution of the determined action;
queuing input information destined for the first procedure upon execution of the determined action; and
submitting said queued input information to said first procedure upon re-starting execution of the at least one of the first procedure and the predetermined project definition.
7. The computer-readable media according to claim 6 , wherein said program is operable when executed to facilitate;
execution within the computer system wherein it comprises at least first and second processors, the first procedure being executed by the first processor and the second procedure being executed by the second processor.
8. The computer-readable media according to claim 6 , wherein said program is operable when executed to facilitate;
that the predetermined performance characteristic relates to at least one of memory utilization of a processor within the computer system, utilization of a processor within the computer system, status of a processor process, utilization of a network forming part of the computer system, and communications activity of a network forming part of the computer system.
9. The computer-readable media according to claim 6 , wherein said program is operable when executed to facilitate;
re-starting execution of at least one of the first procedure and the predetermined project definition is carried out so as to resume execution of one of the at least one of the first procedure and the predetermined project definition which was interrupted, beginning with data from the associated data source which immediately follows the last data therefrom which was processed to completion and stored in the associated data destination.
10. The computer-readable media according to claim 6 , wherein said program is operable when executed to facilitate;
that the first procedure executes the set of predetermined project definitions in a predetermined order upon a set of data files stored within a memory connected to a computer network to which the computer system is also connected.