IP Library Granted Patent US 11,756,158
Granted Patent B2
US 11,756,158 · App. 17/724,181 · Granted Sep 12, 2023

Information processing device, information processing method, information processing program, image processing device, and image processing system for associating position information with captured images

Inventors: Hiroshi Mori (Tokyo, JP); Hiroyuki Sano (Tokyo, JP); Hideo Miyamaki (Tokyo, JP); Masato Tsujie (Tokyo, JP); Yoichiro Sato (Kanagawa, JP)
Assignee: SONY GROUP CORPORATION
G06T3/4038H04N23/662H04N23/695
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,756,158
App. No.
17/724,181
Granted
Sep 12, 2023
Kind
B2
Abstract

An information processing device configured to: acquire an image captured according to an imaging instruction by a mobile object; acquire sensor information including position information about the mobile object according to an exposure notification signal transmitted at a fixed timing that is relative to an exposure end timing of the captured image, the exposure notification signal being transmitted in response to the imaging instruction; and store the position information included in the acquired sensor information in association with the acquired image.

Claims (32)

1. An information processing device, comprising:

processing circuitry configured to

acquire an image captured according to an imaging instruction by a mobile object,

acquire sensor information including position information about the mobile object according to an exposure notification signal transmitted at a fixed timing that is relative to an exposure end timing of the captured image, the exposure notification signal being transmitted in response to the imaging instruction, and

store the position information included in the acquired sensor information in association with the acquired image.

2. The information processing device according to claim 1 , wherein the processing circuitry is configured to acquire the image captured according to the imaging instruction by an imaging device mounted on the mobile object.

3. The information processing device according to claim 2 , wherein the processing circuitry is configured to acquire the sensor information including the position information about the mobile object from a sensor device according to the exposure notification signal transmitted by the imaging device to the sensor device at the fixed timing that is relative to the exposure end timing of the captured image, the exposure notification signal being transmitted from the imaging device to the sensor device in response to the imaging instruction.

4. The information processing device according to claim 1 , wherein the exposure notification signal indicates the exposure end timing related to the capture of the image by an imaging device.

5. The information processing device according to claim 4 , wherein the sensor information is most recent sensor information at a point in time of the exposure end timing.

6. The information processing device according to claim 4 , wherein the fixed timing that is relative to the exposure end timing is the exposure end timing of an end of the exposure that the imaging device performs to capture the image, a timing of a beginning of the exposure that the imaging device performs to capture the image, or a timing at which the imaging instruction for the imaging device to capture the image is issued.

7. The information processing device according to claim 1 , wherein the processing circuitry is further configured to acquire elevation data indicating an elevation of a ground surface imaged by an imaging device, and associate the elevation data with the sensor information.

8. The information processing device according to claim 7 , wherein the processing circuitry is further configured to associate the elevation data with the image.

9. The information processing device according to claim 7 , wherein the elevation data is acquired from map data containing information about a position and an elevation, on a basis of the position information about the mobile object.

10. The information processing device according to claim 7 , wherein the elevation data is acquired on a basis of distance information acquired by a ranging sensor mounted on the mobile object.

11. The information processing device according to claim 10 , wherein the processing circuitry is further configured to acquire the elevation data from the ranging sensor according to the exposure notification signal transmitted from the imaging device in response to the imaging instruction, and associate the acquired elevation data with the sensor information.

12. An information processing method, comprising:

acquiring an image captured according to an imaging instruction by a mobile object;

acquiring sensor information including position information about the mobile object according to an exposure notification signal transmitted at a fixed timing that is relative to an exposure end timing of the captured image, the exposure notification signal being transmitted in response to the imaging instruction; and

storing the position information included in the acquired sensor information in association with the acquired image.

13. The information processing method according to claim 12 , wherein the acquiring acquires the image captured according to the imaging instruction by an imaging device mounted on the mobile object.

14. The information processing method according to claim 13 , wherein the acquiring acquires the sensor information including the position information about the mobile object from a sensor device according to the exposure notification signal transmitted by the imaging device to the sensor device at the fixed timing that is relative to the exposure end timing of the captured image, the exposure notification signal being transmitted from the imaging device to the sensor device in response to the imaging instruction.

15. An image processing device, comprising:

processing circuitry configured to

receive a plurality of images with sensor information including position information, each position information being stored in association with a different one of the plurality of images and about a mobile object acquired according to an exposure notification signal transmitted at a fixed timing that is relative to an exposure end timing of the respective one of the plurality of images, the exposure notification signal being transmitted in response to an imaging instruction, and

generate a composite image by arranging the plurality of images on a basis of the position information included in the sensor information.

16. The image processing device according to claim 15 , wherein the processing circuitry is configured to receive a plurality of images with sensor information including position information, each position information being stored in association with a different one of the plurality of images and about a mobile object acquired according to an exposure notification signal transmitted by an imaging device to a sensor device at a fixed timing that is relative to an exposure end timing of the respective one of the plurality of images, the exposure notification signal being transmitted from the imaging device to the sensor device in response to an imaging instruction.

17. The image processing device according to claim 15 , wherein the processing circuitry is further configured to generate the composite image by arranging the plurality of images on a basis of elevation data indicating an elevation of a ground surface imaged by an imaging device.

18. The image processing device according to claim 17 , wherein the processing circuitry is further configured to define a first coordinate system corresponding to an imaging range of the imaging device in association with a movement of the mobile object and a second coordinate system corresponding to a plane in which one of the plurality of images is arranged,

compute a projected position of the one of the plurality of images on the ground surface in the first coordinate system on a basis of the sensor information and the elevation data, and

arrange the one of the plurality of images by converting the projected position to the second coordinate system.

19. The image processing device according to claim 17 , wherein the elevation data is acquired from map data containing information about a position and an elevation.

20. The image processing device according to claim 17 , wherein the elevation data is acquired by a ranging sensor mounted on the mobile object.

Assignments (1)
CHANGE OF NAME Recorded Jan 27, 2023
From: SONY CORPORATION
To: SONY GROUP CORPORATION
Reel/Frame 062525/0545 →
Priority Claims (2)
JP 2017-090763 · Apr 28, 2017 · national
JP 2018-024840 · Feb 15, 2018 · national
Continuity (2)
Continuation 16607006
Related Publication 20220237738A1 · Jul 28, 2022