Fusion of spatial and temporal context for location determination for visualization systems
A computer-implemented method for generating a control signal by locating at least one instrument by way of a combination of machine learning systems on the basis of digital images is described. In this case, the method includes determining parameter values of a movement context by using the at least two digital images and determining an influence parameter value which controls an influence of one of the digital images and the parameter values of the movement context on the input data which are used within a first trained machine learning system, which has a first learning model, for generating the control signal.
1 . A computer-implemented method for generating a control signal by locating at least one instrument by way of a combination of machine learning systems on the basis of digital images, the method comprising:
providing at least two digital images of the same spatial scene with a movement of the instrument in the scene as input data;
determining parameter values of a movement context by using the at least two digital images; and
determining an influence parameter value, which controls the influence of:
one of the digital images, and
the parameter values of the movement context on the input data which are used within a first trained machine learning system, which has a first learning model, for generating the control signal,
wherein the first trained machine learning system comprises:
a second machine learning system which was trained to generate output values in the form of a first feature tensor from at least one digital image;
a third machine learning system which was trained for generating output values in the form of a second feature tensor from the parameter values of the movement context; and
a weight unit adapted to control the influence of the first feature tensor vis-à-vis the influence of the second feature tensor on a fourth machine learning system in the first trained machine learning system.
2 . The method of claim 1 , wherein the control signal is adapted to control a robotic visualization system.
3 . The method of claim 1 , wherein the influence parameter value is determined by extracting parameter values of an image property from at least one of the at least two digital images.
4 . The method of claim 3 , wherein the parameter values of the image property are represented by at least one image property selected from a group comprising: an image unsharpness map; an image contrast map; an image color saturation map; an image color homogeneity map; an indicator value for specular reflection zones; an image brightness map; a shadow effect indicator value; a masking index value; and an image artifact index value, in each case related to the at least one image.
5 . The method of claim 3 , wherein:
at least one of the at least two digital images;
the parameter values of the movement context; and
the parameter values of the image property are used as input values for the first trained machine learning system.
6 . The method of claim 1 , further comprising:
increasing the influence of the first feature tensor vis-à-vis the second feature tensor if an image property value is higher than a given threshold; and
increasing the influence of the second feature tensor vis-à-vis the first feature tensor if the image property value is lower than a given threshold.
7 . The method of claim 6 , wherein:
in optically sharp image regions, the second feature tensor is over-weighted vis-à-vis the first feature tensor; and
in optically blurred image regions, the first feature tensor is over-weighted vis-à-vis the second feature tensor.
8 . A computer-implemented method for generating a control signal by locating at least one instrument by way of a combination of machine learning systems on the basis of digital images, the method comprising:
providing at least two digital images of the same spatial scene with a movement of the instrument in the scene as input data;
determining parameter values of a movement context by using the at least two digital images; and
determining an influence parameter value, which controls the influence of:
one of the digital images; and
the parameter values of the movement context on the input data which are used within a first trained machine learning system, which has a first learning model, for generating the control signal,
wherein the first trained machine learning system comprises:
a second machine learning system which was trained to generate values of a first feature tensor and a first uncertainty value from at least one digital image; and
a third machine learning system which was trained to generate values of a second feature tensor and a second uncertainty value from the parameter values of the movement context, with the first feature tensor and the first uncertainty value and the second feature tensor and the second uncertainty value serving as input data for a fourth machine learning system which was trained to generate the control signal.
9 . The method of claim 8 , wherein the first trained machine learning system additionally comprises a weight unit which controls the influence of the first feature tensor vis-à-vis the influence of the second feature tensor on the fourth machine learning system.
10 . The method of claim 9 , further comprising:
increasing the influence of the first feature tensor vis-à-vis the second feature tensor if the second uncertainty value is higher than the first uncertainty value, and
increasing the influence of the second feature tensor vis-à-vis the first feature tensor if the first uncertainty value is higher than the second uncertainty value.
11 . The method of claim 8 , wherein either the first uncertainty value or the second uncertainty value is zero.
12 . The method of claim 8 , wherein uncertainty values are determined by an ensemble learning method.
13 . A control system for generating a control signal by locating at least one instrument by way of a combination of machine learning systems on the basis of digital images, the control system comprising:
a processor and a memory connected to the processor, the memory storing program code segments which, when executed by the processor, prompt the processor to perform operations comprising:
receiving at least two digital images of the same spatial scene with a movement of the instrument in the scene as input data;
determining parameter values of a movement context by using the at least two digital images; and
determining an influence parameter value, which controls the influence of:
one of the digital images; and
the parameter values of the movement context on the input data which are used within a first trained machine learning system, which has a first learning model, for generating the control signal,
wherein the first trained machine learning system comprises one of:
a system comprising:
a second machine learning system which was trained to generate output values in the form of a first feature tensor from at least one digital image;
a third machine learning system which was trained for generating output values in the form of a second feature tensor from the parameter values of the movement context; and
a weight unit adapted to control the influence of the first feature tensor vis-à-vis the influence of the second feature tensor on a fourth machine learning system in the first trained machine learning system; and
a system comprising:
a second machine learning system which was trained to generate values of a first feature tensor and a first uncertainty value from at least one of the digital images; and
a third machine learning system which was trained to generate values of a second feature tensor and a second uncertainty value from the parameter values of the movement context, with the first feature tensor and the first uncertainty value and the second feature tensor and the second uncertainty value serving as input data for a fourth machine learning system which was trained to generate the control signal.
14 . The control system of claim 13 , wherein the control signal is adapted to control a robotic visualization system.
15 . The control system of claim 13 , wherein the influence parameter value is determined by extracting parameter values of an image property from at least one of the at least two digital images.
16 . The control system of claim 15 , wherein the parameter values of the image property are represented by at least one image property selected from a group comprising: an image unsharpness map; an image contrast map; an image color saturation map; an image color homogeneity map; an indicator value for specular reflection zones; an image brightness map; a shadow effect indicator value; a masking index value; and an image artifact index value, in each case related to the at least one image.
17 . The control system of claim 15 , wherein:
at least one of the at least two digital images;
the parameter values of the movement context; and
the parameter values of the image property are used as input values for the first trained machine learning system.
18 . The control system of claim 13 , wherein the operations further comprise:
increasing the influence of the first feature tensor vis-à-vis the second feature tensor if an image property value is higher than a given threshold, and
increasing the influence of the second feature tensor vis-à-vis the first feature tensor if the image property value is lower than a given threshold.
19 . The control system of claim 18 , wherein:
in optically sharp image regions, the second feature tensor is over-weighted vis-à-vis the first feature tensor, and
in optically blurred image regions, the first feature tensor is over-weighted vis-à-vis the second feature tensor.