Modifying parameter sets characterising a computer vision model
Modifying a visual parameter specification characterising the operational design domain of the computer vision model by improving the visual parameter specification according to a sensitivity analysis of the computer vision model.
1. A computer-implemented method for modifying a first visual parameter specification to provide a second visual parameter specification characterising a computer vision model, the method comprising the following steps:
obtaining a computer vision model configured to perform a computer vision function characterising elements of observed scenes;
obtaining a first visual parameter specification including at least one initial visual parameter set, wherein an item of visual data provided based on an extent of the at least one visual parameter set is capable of affecting a classification or regression performance of the computer vision model;
providing a visual data set includes a subset of items of visual data compliant with the first visual parameter specification, and a corresponding subset of items of groundtruth data;
applying the subset of items of visual data to the computer vision model to obtain a plurality of performance scores characterizing performance of the computer vision model when applied to the subset of items of visual data of the visual data set, using the corresponding groundtruth data;
performing a sensitivity analysis of the plurality of performance scores over a domain of the first visual parameter specification; and
generating a second visual parameter specification including at least one updated visual parameter set, wherein the at least one initial visual parameter set is modified based on an outcome of the sensitivity analysis to provide the at least one updated visual parameter set.
2. The computer-implemented method according to claim 1 , wherein obtaining the plurality of performance scores further comprises:
generating, using the computer vision model, a plurality of predictions of elements of observed scenes in the subset of items of visual data, wherein the plurality of predictions include at least one prediction of a classification label and/or at least one regression value of at least one item in the subset of visual data; and
comparing the plurality of predictions of elements in the subset of items of visual data with the corresponding subset of groundtruth data, to obtain the plurality of performance scores.
3. The computer-implemented method according to claim 1 , further comprising:
identifying, based on an identification condition, the at least one initial visual parameter set of the first visual parameter specification using the plurality of performance scores, and wherein the generating of the second visual parameter specification includes modifying the at least one initial visual parameter set by dividing the at least one initial visual parameter set into at least a first and a second modified visual parameter set, or combining a first and a second visual parameter set into a cluster.
4. The computer-implemented method according to claim 1 , further comprising:
identifying, the at least one initial visual parameter set of the first visual parameter specification using the plurality of performance scores, and wherein the generating of the second visual parameter specification includes modifying the at least one initial visual parameter set by enlarging or shrinking a scope of the at least one initial visual parameter set on its domain to yield a modified visual parameter set.
5. The computer-implemented method according to claim 1 , wherein the performing of the sensitivity analysis includes:
computing a plurality of variances of respective performance scores of the plurality of performance scores with respect to the initial visual parameters of the first visual parameter specification; and
ranking the initial visual parameters of the first visual parameter specification based on the computed plurality of variances.
6. The computer-implemented method according to claim 1 , wherein the providing of the visual data set compliant with the first visual parameter specification includes:
sampling the at least one initial visual parameter set included in the first visual parameter set to obtain a set of sampled initial visual parameter values; and
obtaining the visual data set of initial visual parameter values using the set of sampled initial visual parameter values.
7. The computer-implemented method according to claim 6 , wherein the sampling of the at least one initial visual parameter set is performed using combinatorial testing, or by Latin hypercube sampling.
8. The computer-implemented method according to claim 1 , wherein the domain of the first visual parameter specification includes a subset, in a finite-dimensional vector space, of numerical representations that visual parameters are allowed to lie in, or a multi-dimensional interval of continuous or discrete visual parameters, or a set of numerical representations of visual parameters in the finite-dimensional vector space.
9. The computer-implemented method according to claim 1 , further comprising:
verifying the second visual parameter specification by:
sampling the second visual parameter specification;
providing a further visual dataset comprising a subset of items of visual data compliant with the second visual parameter specification based on the samples of the second visual parameter specification, and a corresponding subset of items of groundtruth data, and
testing the subset of items of visual data using the computer vision model according to a reduced dimensionality input space to provide a verification result of the second visual parameter specification with reduced complexity.
10. The computer-implemented method according to claim 1 , further comprising:
displaying, via a graphical user interface displayed on output interface, a graphical representation of the second visual parameter specification to a user.
11. The computer-implemented method according to claim 10 , wherein the displaying includes displaying the graphical representation of the second visual parameter specification in combination with a graphical representation of the first visual parameter specification.
12. The computer-implemented method according to claim 10 , wherein the displaying includes displaying the graphical representation of the second visual parameter specification in combination with a graphical representation of the outcome of the sensitivity analysis.
13. The computer-implemented method according to claim 10 , further comprising:
generating a third visual parameter specification as a consequence of an interactive user amendment to the graphical representation of the second visual parameter specification via the graphical user interface.
14. A computer-implemented method for providing a set of training data, comprising:
obtaining a second visual parameter specification generated by:
obtaining a computer vision model configured to perform a computer vision function characterising elements of observed scenes,
obtaining a first visual parameter specification including at least one initial visual parameter set, wherein an item of visual data provided based on an extent of the at least one visual parameter set is capable of affecting a classification or regression performance of the computer vision model,
providing a visual data set includes a subset of items of visual data compliant with the first visual parameter specification, and a corresponding subset of items of groundtruth data,
applying the subset of items of visual data to the computer vision model to obtain a plurality of performance scores characterizing performance of the computer vision model when applied to the subset of items of visual data of the visual data set, using the corresponding groundtruth data,
performing a sensitivity analysis of the plurality of performance scores over a domain of the first visual parameter specification, and
generating the second visual parameter specification including at least one updated visual parameter set, wherein the at least one initial visual parameter set is modified based on an outcome of the sensitivity analysis to provide the at least one updated visual parameter set; and
obtaining a training data set, wherein the training data set is obtained by one or a combination of:
generating, using a synthetic visual data generator, a synthetic training data set including synthetic visual data and groundtruth data synthesized according to the second visual parameter set, and/or
sampling items of visual data from a database comprising specimen images associated with corresponding items of groundtruth data according to the second visual parameter set, and/or
specifying experimental requirements according to the second visual parameter set, and performing live experiments to obtain the training data set; and
outputting the training data set.
15. A computer-implemented method for training a computer vision model, comprising:
obtaining a further computer vision model configured to perform a computer vision function characterising elements of observed scenes; and
obtaining a training data set by:
obtaining a second visual parameter specification generated by:
obtaining a computer vision model configured to perform a computer vision function characterising elements of observed scenes,
obtaining a first visual parameter specification including at least one initial visual parameter set, wherein an item of visual data provided based on an extent of the at least one visual parameter set is capable of affecting a classification or regression performance of the computer vision model,
providing a visual data set includes a subset of items of visual data compliant with the first visual parameter specification, and a corresponding subset of items of groundtruth data,
applying the subset of items of visual data to the computer vision model to obtain a plurality of performance scores characterizing performance of the computer vision model when applied to the subset of items of visual data of the visual data set, using the corresponding groundtruth data,
performing a sensitivity analysis of the plurality of performance scores over a domain of the first visual parameter specification, and
generating the second visual parameter specification including at least one updated visual parameter set, wherein the at least one initial visual parameter set is modified based on an outcome of the sensitivity analysis to provide the at least one updated visual parameter set; and
obtaining the training data set by one or a combination of:
generating, using a synthetic visual data generator, a synthetic training data set including synthetic visual data and groundtruth data synthesized according to the second visual parameter set, and/or
sampling items of visual data from a database comprising specimen images associated with corresponding items of groundtruth data according to the second visual parameter set, and/or
specifying experimental requirements according to the second visual parameter set, and performing live experiments to obtain the training data set; and
outputting the training data set;
training the computer vision model using the training data set.
16. An apparatus for modifying a first visual parameter specification to provide a second visual parameter specification characterising a computer vision model, comprising:
an input interface;
a processor;
a memory; and
an output interface;
wherein the input interface is configured to obtain a computer vision model configured to perform a computer vision function characterising elements of observed scenes, and to obtain a first visual parameter specification including at least one initial visual parameter set, wherein generating an item of visual data based on the extent of the at least one visual parameter set is capable of affecting a classification or regression performance of the computer vision model;
wherein the processor is configured to provide a visual data set compliant with the first visual parameter specification, wherein the visual data set includes a subset of items of visual data, and a corresponding subset of items of groundtruth data, to apply the subset of items of visual data to the computer vision model to obtain a plurality of performance scores characterizing the performance of the computer vision model for a plurality of items of visual data and the corresponding groundtruth data, to perform a sensitivity analysis of the plurality of performance scores over a domain of the first visual parameter specification, and to generate a second visual parameter specification comprising at least one updated visual parameter set, wherein the at least one initial visual parameter set is modified based on the outcome of the sensitivity analysis.
17. A non-transitory computer readable medium on which is stored a computer program for modifying a first visual parameter specification to provide a second visual parameter specification characterising a computer vision model, the computer program, when executed by a processor, causing the processor to perform the following steps:
obtaining a computer vision model configured to perform a computer vision function characterising elements of observed scenes;
obtaining a first visual parameter specification including at least one initial visual parameter set, wherein an item of visual data provided based on an extent of the at least one visual parameter set is capable of affecting a classification or regression performance of the computer vision model;
providing a visual data set includes a subset of items of visual data compliant with the first visual parameter specification, and a corresponding subset of items of groundtruth data;
applying the subset of items of visual data to the computer vision model to obtain a plurality of performance scores characterizing performance of the computer vision model when applied to the subset of items of visual data of the visual data set, using the corresponding groundtruth data;
performing a sensitivity analysis of the plurality of performance scores over a domain of the first visual parameter specification; and
generating a second visual parameter specification including at least one updated visual parameter set, wherein the at least one initial visual parameter set is modified based on an outcome of the sensitivity analysis to provide the at least one updated visual parameter set.