Information processing system, method and program

1 . An information processing system comprising:

at least one memory configured to store program code;

at least one processor configured to operate as instructed by the program code, the program code including:

captured image acquisition code configured to cause at least one of the at least one processor to acquire a captured image captured during flight by an imaging device mounted on an airplane;

flight information acquisition code configured to cause at least one of the at least one processor to acquire flight information detected using a sensor mounted on the airplane;

data analysis code configured to cause at least one of the at least one processor to acquire motion information in a time series of the airplane or the imaging device by analyzing a plurality of the captured images continuously captured in a time series and the flight information continuously acquired in a time series;

flight information specification code configured to cause at least one of the at least one processor to specify the flight information acquired simultaneously with or nearly simultaneously with the captured image by matching motion information acquired by analyzing the captured images and motion information acquired by analyzing the flight information to each other;

image correction code configured to cause at least one of the at least one processor to acquire, by correcting the captured image on a basis of information acquired using the flight information, a correction image in which a predetermined direction in a state where the image is erected and a reference direction in the image are matched to each other; and

direction information acquisition code configured to cause at least one of the at least one processor to acquire direction information in the captured image with the earth as a reference, on a basis of information on a posture of the airplane or the imaging device acquired as the flight information,

wherein the image correction code is further configured to cause at least one of the at least one processor to acquire, by correcting the captured image on a basis of the direction information, the correction image in which the predetermined direction in the state where the image is erected and the reference direction in the image are matched to each other,

wherein the direction information acquisition code is further configured to cause at least one of the at least one processor to acquire the direction information on a basis of the flight information acquired simultaneously with or nearly simultaneously with the captured image,

wherein the direction information acquisition code is further configured to cause at least one of the at least one processor to acquire the direction information on a basis of specified flight information,

wherein the flight information specification code is further configured to cause at least one of the at least one processor to specify the flight information acquired simultaneously with or nearly simultaneously with the captured image by interpolating, based on a machine learning model or a statistical model trained using previously acquired flight information, a first sequence of a parameter included in motion information acquired by analyzing the captured images and a second sequence of the parameter included in motion information acquired by analyzing the flight information, and generating a sequence alignment representing correspondence between the sequences by comparing the interpolated first sequence and the second sequence, and performing matching based on the generated sequence alignment.

2 . The information processing system according to claim 1 , wherein

the image correction code is further configured to cause at least one of the at least one processor to acquire, by rotating the captured image on a basis of the direction information, the correction image in which the predetermined direction in the state where the image is erected and a predetermined compass direction in the image are matched to each other.

3 . The information processing system according to claim 1 , wherein

the image correction code is further configured to cause at least one of the at least one processor to acquire, by rotating the captured image on a basis of the direction information, the correction image in which the predetermined direction in the state where the image is erected and a north in the image are matched to each other.

4 . The information processing system according to claim 1 , wherein

the image correction code is further configured to cause at least one of the at least one processor to acquire, by rotating the captured image on a basis of the direction information, the correction image in which the predetermined direction in the state where the image is erected and a vertical direction in the image are matched to each other.

5 . The information processing system according to claim 1 , wherein

the flight information acquisition code is further configured to cause at least one of the at least one processor to acquire, as the flight information, three-dimensional information detected by at least one type of three-axis sensor mounted on the airplane.

6 . The information processing system according to claim 1 , further comprising:

data analysis code configured to cause at least one of the at least one processor to acquire motion information in a time series of the airplane or the imaging device by analyzing a plurality of the captured images continuously captured in a time series or the flight information continuously acquired in a time series; and

captured image selection code configured to cause at least one of the at least one processor to select, on a basis of motion information acquired by the data analysis unit, a captured image in which a state of the imaging device during imaging is close to a stationary state, wherein

the image correction code is further configured to cause at least one of the at least one processor to acquire the correction image by correcting the selected captured image.

7 . The information processing system according to claim 1 , further comprising:

satellite image acquisition code configured to cause at least one of the at least one processor to acquire a satellite image corresponding to the captured image on a basis of positional information showing a flight position of the airplane, the positional information being acquired as the flight information; and

an image comparison code configured to cause at least one of the at least one processor to detect, by detecting and comparing a characteristic in the satellite image and a characteristic in the captured image with each other, a corresponding characteristic captured in both the satellite image and the captured image, wherein

the image correction code is further configured to cause at least one of the at least one processor to acquire, by correcting the captured image so that a position of the corresponding characteristic in the image is matched or approximated to a position of the corresponding characteristic in the satellite image, the correction image in which the predetermined direction in the state where the image is erected and the reference direction in the image are matched to each other.

8 . The information processing system according to claim 7 , further comprising:

perspective correction code configured to cause at least one of the at least one processor to correct the satellite image and/or the captured image so that perspectives of the satellite image and the captured image are matched or approximated to each other, wherein

the image comparison code is further configured to cause at least one of the at least one processor to detect the corresponding characteristic using the satellite image and/or the captured image after correction by the perspective correction unit.

9 . The information processing system according to claim 8 , wherein

the perspective correction code configured to cause at least one of the at least one processor to estimate a distance from an imaging point to an imaging object related to the captured image and correct the satellite image and/or the captured image on a basis of the estimated distance.

10 . The information processing system according to claim 8 , wherein

the perspective correction code is further configured to cause at least one of the at least one processor to estimate a shape of an imaging object in the captured image and corrects the satellite image and/or the captured image on a basis of the estimated shape.

11 . An information processing method, performed by at least one processor and comprising:

acquiring a captured image captured during flight by an imaging device mounted on an airplane;

acquiring flight information detected using a sensor mounted on the airplane;

acquiring motion information in a time series of the airplane or the imaging device by analyzing a plurality of the captured images continuously captured in a time series and the flight information continuously acquired in a time series;

specifying the flight information acquired simultaneously with or nearly simultaneously with the captured image by matching motion information acquired by analyzing the captured images and motion information acquired by analyzing the flight information to each other;

acquiring, by correcting the captured image on a basis of information acquired using the flight information, a correction image in which a predetermined direction in a state where the image is erected and a reference direction in the image are matched to each other;

acquiring direction information in the captured image with the earth as a reference, on a basis of information on a posture of the airplane or the imaging device acquired as the flight information;

acquiring, by correcting the captured image on a basis of the direction information, the correction image in which the predetermined direction in the state where the image is erected and the reference direction in the image are matched to each other;

acquiring the direction information on a basis of the flight information acquired simultaneously with or nearly simultaneously with the captured image;

acquiring the direction information on a basis of specified flight information;

specifying the flight information acquired simultaneously with or nearly simultaneously with the captured image by interpolating, based on a machine learning model or a statistical model trained using previously acquired flight information, a first sequence of a parameter included in motion information acquired by analyzing the captured images and a second sequence of the parameter included in motion information acquired by analyzing the flight information;

generating a sequence alignment representing correspondence between the sequences by comparing the interpolated first sequence and the second sequence; and

performing matching based on the generated sequence alignment.

12 . A non-transitory computer readable medium storing a computer program that when executed by at least one processor, causes the at least one processor to:

acquire a captured image captured during flight by an imaging device mounted on an airplane;

acquire flight information detected using a sensor mounted on the airplane;

acquire motion information in a time series of the airplane or the imaging device by analyzing a plurality of the captured images continuously captured in a time series and the flight information continuously acquired in a time series;

specify the flight information acquired simultaneously with or nearly simultaneously with the captured image by matching motion information acquired by analyzing the captured images and motion information acquired by analyzing the flight information to each other;

acquire, by correcting the captured image on a basis of information acquired using the flight information, a correction image in which a predetermined direction in a state where the image is erected and a reference direction in the image are matched to each other;

acquire direction information in the captured image with the earth as a reference, on a basis of information on a posture of the airplane or the imaging device acquired as the flight information;

acquire, by correcting the captured image on a basis of the direction information, the correction image in which the predetermined direction in the state where the image is erected and the reference direction in the image are matched to each other;

acquire the direction information on a basis of the flight information acquired simultaneously with or nearly simultaneously with the captured image;

acquire the direction information on a basis of specified flight information;

specify the flight information acquired simultaneously with or nearly simultaneously with the captured image by interpolating, based on a machine learning model or a statistical model trained using previously acquired flight information, a first sequence of a parameter included in motion information acquired by analyzing the captured images and a second sequence of the parameter included in motion information acquired by analyzing the flight information;

generate a sequence alignment representing correspondence between the sequences by comparing the interpolated first sequence and the second sequence; and

perform matching based on the generated sequence alignment.