Information presentation device, stereo camera system, and information presentation method

1. An information presentation device comprising a processor that, when executing instructions stored in a memory, performs operations comprising:

acquiring, at predetermined intervals, a first image captured by a first imaging device and a second image captured by a second imaging device, the first imaging device mounted in a first chassis and the second imaging device mounted in a second chassis which is distinct from the first chassis, such that the first imaging device and the second imaging device are movable independently and separately from one another, wherein the first imaging device and second imaging device comprise a stereo camera which recovers three-dimensional position information based on the first image and the second image;

extracting, at the predetermined intervals, feature points from the first image and the second image;

obtaining, at the predetermined intervals, a parameter value based on the extracted feature points;

determining, at the predetermined intervals, a change between the currently obtained parameter value and the previously obtained parameter value;

determining, at the predetermined intervals, whether or not an abnormality is present in a relative positional relationship or a relative relationship of orientation between the first imaging device and the second imaging device according to the determined change in the parameter value; and

presenting warning information including an instruction that there is an abnormality to correct at least one of the position and orientation of one of the first imaging device and the second imaging device by physically moving at least one of the first imaging device and the second imaging device, when it is determined that the abnormality is present,

wherein, in the presenting, a plurality of parameter values obtained at the predetermined intervals are displayed to be arranged along a time axis, together with a predetermined threshold.

2. The information presentation device according to claim 1 ,

wherein the parameter value is obtained by counting a number of the extracted feature points, and

it is determined whether or not an abnormality is present based on the counted number of extracted feature points.

3. The information presentation device according to claim 1 ,

wherein the parameter value is obtained by detecting matching points that match at least one feature point, of the extracted feature points, which is included in the first image and at least one feature point, of the extracted feature points, which is included in the second image,

it is determined whether or not the abnormality is present based on a number of detected matching points.

4. The information presentation device according to claim 1 ,

wherein, in determining the parameter value, the processor further performs operations comprising:

detecting matching points that match a plurality of first feature points, of the extracted feature points, which are included in the first image and a plurality of second feature points, of the extracted feature points, which are included in the second image; and

estimating at least one of a position and an orientation of the second imaging device with respect to the first imaging device based on the detected matching points,

wherein it is determined that the abnormality is present based on a change in at least one of the estimated position and the orientation of the second imaging device with respect to the first imaging device.

5. The information presentation device according to claim 1 ,

wherein the warning information includes information on a coping method according to a cause of the abnormality.

6. An information presentation device comprising a processor, executing instructions stored in a memory performs operations comprising:

acquiring a first image captured by a first imaging device and a second image captured by a second imaging device, the first imaging device mounted in a first chassis and the second imaging device mounted in a second chassis which is distinct from the first chassis, such that the first imaging device and the second imaging device are movable independently and separately from one another, wherein the first imaging device and second imaging device comprise a stereo camera which recovers three-dimensional position information based on the first image and the second image;

extracting feature points from the first image and the second image;

detecting matching points that match a plurality of first feature points, of the extracted feature points, which are included in the first image and a plurality of second feature points, of the extracted feature points, which are included in the second image;

estimating at least one of a position and an orientation of the second imaging device with respect to the first imaging device based on the detected matching points;

determining whether or not an abnormality is present in a relative positional relationship or a relative relationship of orientation between the first imaging device and the second imaging device based on a change in at least one of the estimated position and the orientation of the second imaging device with respect to the first imaging device;

presenting warning information including an instruction that there is an abnormality to correct at least one of the position and orientation of one of the first imaging device and the second imaging device by physically moving at least one of the first imaging device and the second imaging device, when it is determined that the abnormality is present;

determining three-dimensional coordinates of target points, which are obtained by reverse projecting two dimensional coordinates of the plurality of first feature points in the first image that are included in the detected matching points; and

determining whether or not a variation in the determined three-dimensional coordinates of the target points is equal to or less than a predetermined value,

wherein, in the presenting, information indicating that reliability of the estimated position and the orientation of the second imaging device with respect to the first imaging device is low, when it is determined that the variation is equal to or less than the predetermined value.

7. The information presentation device according to claim 6 , wherein the variation in the determined three-dimensional coordinates of the target points being one of a standard deviation or a dispersion.

8. A stereo camera system comprising:

a first imaging device that captures a first image and is fixed to a first chassis, and

a second imaging device that captures a second image and is fixed to a second chassis which is distinct from the first chassis, such that the first imaging device and the second imaging device are movable independently and separately from one another, wherein the first imaging device and second imaging device comprise a stereo camera which recovers three-dimensional position information based on the first image and the second image;

an information presentation device that is connected to the first imaging device and the second imaging device via a network, and comprises a processor that, when executing instructions stored in a memory, performs operations comprising:

acquiring, at predetermined intervals, the first image captured by the first imaging device and the second image captured by the second imaging device via the network;

extracting, at the predetermined intervals, feature points from the first image and the second image;

obtaining, at the predetermined intervals, a parameter value based on the extracted feature points;

determining, at the predetermined intervals, a change between the currently obtained parameter value and the previously obtained parameter value;

determining, at the predetermined intervals, whether or not an abnormality is present in a relative positional relationship or a relative relationship of orientation between the first imaging device and the second imaging device according to the determined change in the parameter value, and

presenting warning information including an instruction that there is the abnormality to correct at least one of the position and orientation of one of the first imaging device and the second imaging device by physically moving at least one of the first imaging device and the second imaging device, when it is determined that the abnormality is present,

wherein, in the presenting, a plurality of parameter values obtained at the predetermined intervals are displayed to be arranged along a time axis, together with a predetermined threshold.

9. An information presentation method comprising:

acquiring, at predetermined intervals, a first image captured by a first imaging device and a second image captured by a second imaging device, the first imaging device mounted in a first chassis and the second imaging device mounted in a second chassis which is distinct from the first chassis, such that the first imaging device and the second imaging device are movable independently and separately from one another, wherein the first imaging device and second imaging device comprise a stereo camera which recovers three-dimensional position information based on the first image and the second image;

extracting, at the predetermined intervals, feature points from the first image and the second image;

obtaining, at the predetermined intervals, a parameter value based on the extracted feature points;

determining, at the predetermined intervals, a change between the currently obtained parameter value and the previously obtained parameter value;

determining, at the predetermined intervals, whether or not an abnormality is present in a relative positional relationship or a relative relationship of orientation between the first imaging device and the second imaging device according to the determined change in the parameter value; and

presenting warning information including an instruction that there is the abnormality to correct at least one of the position and orientation of one of the first imaging device and the second imaging device by physically moving at least one of the first imaging device and the second imaging device, when it is determined that the abnormality is present,

wherein, in the presenting, a plurality of parameter values obtained at the predetermined intervals are displayed to be arranged along a time axis, together with a predetermined threshold.

10. A stereo camera system comprising:

a first imaging device that captures a first image and is fixed to a first chassis, and

a second imaging device that captures a second image and is fixed to a second chassis which is distinct from the first chassis, such that the first imaging device and the second imaging device are movable independently and separately from one another, wherein the first imaging device and second imaging device comprise a stereo camera which recovers three-dimensional position information based on the first image and the second image;

an information presentation device that is connected to the first imaging device and the second imaging device via a network, and comprises a processor that, when executing instructions stored in a memory, performs operations comprising:

acquiring, the first image captured by the first imaging device and the second image captured by the second imaging device via the network;

extracting feature points from the first image and the second image;

detecting matching points that match a plurality of first feature points, of the extracted feature points, which are included in the first image and a plurality of second feature points, of the extracted feature points, which are included in the second image;

estimating at least one of a position and an orientation of the second imaging device with respect to the first imaging device based on the detected matching points,

determining whether or not an abnormality is present in a relative positional relationship or a relative relationship of orientation between the first imaging device and the second imaging device based on a change in at least one of the estimated position and the orientation of the second imaging device with respect to the first imaging device;

presenting warning information including an instruction that there is an abnormality to correct at least one of the position and orientation of one of the first imaging device and the second imaging device by physically moving at least one of the first imaging device and the second imaging device, when it is determined that the abnormality is present;

determining three-dimensional coordinates of target points, which are obtained by reverse projecting two dimensional coordinates of the plurality of first feature points in the first image that are included in the detected matching points; and

determining whether or not a variation in the determined three-dimensional coordinates of the target points is equal to or less than a predetermined value,

wherein, in the presenting, information indicating that reliability of the estimated position and the orientation of the second imaging device with respect to the first imaging device is low, when it is determined that the variation is equal to or less than the predetermined value.

11. An information presentation method comprising:

acquiring a first image captured by a first imaging device and a second image captured by a second imaging device, the first imaging device mounted in a first chassis and the second imaging device mounted in a second chassis which is distinct from the first chassis, such that the first imaging device and the second imaging device are movable independently and separately from one another, wherein the first imaging device and second imaging device comprise a stereo camera which recovers three-dimensional position information based on the first image and the second image;

extracting feature points from the first image and the second image;

detecting matching points that match a plurality of first feature points, of the extracted feature points, which are included in the first image and a plurality of second feature points, of the extracted feature points, which are included in the second image;

estimating at least one of a position and an orientation of the second imaging device with respect to the first imaging device based on the detected matching points;

determining whether or not an abnormality is present in a relative positional relationship or a relative relationship of orientation between the first imaging device and the second imaging device based on a change in at least one of the estimated position and the orientation of the second imaging device with respect to the first imaging device;

presenting warning information including an instruction that there is an abnormality to correct at least one of the position and orientation of one of the first imaging device and the second imaging device by physically moving at least one of the first imaging device and the second imaging device, when it is determined that the abnormality is present;

determining three-dimensional coordinates of target points, which are obtained by reverse projecting two dimensional coordinates of the plurality of first feature points in the first image that are included in the detected matching points; and

determining whether or not a variation in the determined three-dimensional coordinates of the target points is equal to or less than a predetermined value,

in the presenting, information indicating that reliability of the estimated position and the orientation of the second imaging device with respect to the first imaging device is low, when it is determined that the variation is equal to or less than the predetermined value.