IP Library Granted Patent US 12,243,172
Granted Patent B2
US 12,243,172 · App. 17/823,668 · Granted Mar 4, 2025

Image processing apparatus and image processing method

Inventors: Masaki Fukuchi (Tokyo, JP); Kenichirou Ooi (Kanagawa, JP); Tatsuki Kashitani (Tokyo, JP)
Assignee: SONY GROUP CORPORATION
G06T19/006G06F3/011G06F18/24G06T7/73G06V10/757G06V20/20H04N7/18G06T2207/10016
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,243,172
App. No.
17/823,668
Granted
Mar 4, 2025
Kind
B2
Abstract

An information processing system that acquires image data captured by an image capturing device; identifies a density of distribution of a plurality of feature points in the acquired image data; and controls a display to display guidance information based on the density of the distribution of the plurality of feature points.

Claims (49)

1. An information processing apparatus comprising:

circuitry configured to:

receive an image of a real space captured by an image capturing device,

detect a feature point based on the image,

determine a relative position of an object within the real space based on the image and the feature point, and

generate environment information based on the image and the feature point,

determine an explored area or an un-explored area based on the environment information, and

control a display to

display a first virtual object that provides a navigation instruction that instructs a user to explore the un-explored area containing less environment information than that of the explored area, or

display a second virtual object that provides a navigation instruction that instructs a user to avoid activities outside the explored area.

2. The information processing apparatus of claim 1 , wherein the first virtual object is displayed in an exploration mode.

3. The information processing apparatus of claim 2 , wherein the exploration mode is executed based on a user operation information.

4. The information processing apparatus of claim 1 , wherein the first virtual object or the second virtual object is a dedicated virtual object.

5. The information processing apparatus of claim 1 , wherein the first virtual object or the second virtual object includes at least a graphic or text.

6. The information processing apparatus of claim 1 , wherein the environmental information is generated with a registration time information.

7. The information processing apparatus of claim 1 , wherein the at least one circuitry is further configured to:

control the display to display a third virtual object indicating a boundary between the explored area and the unexplored area.

8. The information processing apparatus of claim 1 , wherein the display is a LCD (Liquid Crystal Display).

9. An information processing method comprising:

receiving, by circuitry, an image of a real space captured by an image capturing device;

detecting, by the circuitry, a feature point based on the image;

determining, by the circuitry, a relative position of an object within the real space based on the image and the feature point;

generating, by the circuitry, environment information based on the image and the feature point;

determining, by the circuitry, an explored area or an un-explored area based on the environment information; and

controlling, by the circuitry a display to display a first virtual object that provides a navigation instruction that instructs a user to explore the un-explored area containing less environment information than that of the explored area, or display a second virtual object that provides a navigation instruction that instructs a user to avoid activities outside the explored area.

10. The information processing method of claim 9 , wherein the first virtual object is displayed in an exploration mode.

11. The information processing method of claim 10 , wherein the exploration mode is executed based on a user operation information.

12. The information processing method of claim 9 , wherein the first virtual object or the second virtual object is a dedicated virtual object.

13. The information processing method of claim 9 , wherein the first virtual object or the second virtual object includes at least a graphic or text.

14. The information processing method of claim 9 , wherein the environmental information is generated with a registration time information.

15. The information processing method of claim 9 , further comprising:

controlling the display to display a third virtual object indicating a boundary between

the explored area and the unexplored area.

16. The information processing method of claim 9 , wherein the display is a LCD (Liquid Crystal Display).

17. A non-transitory computer-readable medium storing instructions that when executed by at least one processor, cause the at least one processor to perform an information processing method comprising:

receiving, by circuitry, an image of a real space captured by an image capturing device;

detecting, by the circuitry, a feature point based on the image;

determining, by the circuitry, a relative position of an object within the real space based on the image and the feature point;

generating, by the circuitry, environment information based on the image and the feature point;

determining, by the circuitry, an explored area or an un-explored area based on the environment information; and

controlling, by the circuitry, a display to display a first virtual object that provides a navigation instruction that instructs a user to explore the un-explored area containing less environment information than that of the explored area, or display a second virtual object that provides a navigation instruction that instructs a user to avoid activities outside the explored area.

18. The non-transitory computer-readable medium of claim 17 , wherein the first virtual object is displayed in an exploration mode.

19. The non-transitory computer-readable medium of claim 18 , wherein the exploration mode is executed based on a user operation information.

20. The non-transitory computer-readable medium of claim 17 , wherein the first virtual object or the second virtual object is a dedicated virtual object.

21. The non-transitory computer-readable medium of claim 17 , wherein the first virtual object or the second virtual object includes at least a graphic or text.

22. The non-transitory computer-readable medium of claim 17 , wherein the environmental information is generated with a registration time information.

23. The non-transitory computer-readable medium of claim 17 , further comprising:

controlling the display to display a third virtual object indicating a boundary between the explored area and the unexplored area.

24. The non-transitory computer-readable medium of claim 17 , wherein the display is a LCD (Liquid Crystal Display).

Assignments (1)
CHANGE OF NAME Recorded Aug 2, 2024
From: SONY CORPORATION
To: SONY GROUP CORPORATION
Reel/Frame 068291/0179 →
Priority Claims (1)
JP 2012-097714 · Apr 23, 2012 · national
Continuity (6)
Continuation 17181051 · Feb 22, 2021
Continuation 15816500 · Nov 17, 2017
Continuation 15384754 · Dec 20, 2016
Continuation 15162246 · May 23, 2016
Continuation 14391874
Related Publication 20220414993A1 · Dec 29, 2022
References Cited (30)
US 20040167667A1 · Goncalves et al. · 2004 [cited by applicant]
US 20090243889A1 · Suhr et al. · 2009 [cited by applicant]
US 20120146998A1 · Kim et al. · 2012 [cited by applicant]
US 20130215230A1 · Miesnieks et al. · 2013 [cited by applicant]
US 20140350839A1 · Pack et al. · 2014 [cited by applicant]
DE 102009012435A1 · 2010 [cited by applicant]
JP 2009237845A · 2009 [cited by applicant]
JP 2011159162A · 2011 [cited by applicant]
KR 1020090103165A · 2009 [cited by applicant]
Doroodgar, Barzin. A learning-based semi-autonomous control architecture for robotic exploration of search and rescue environments. 2011. Library and Archives Canada. pp. 1-80 (Year: 2011). [cited by examiner]
Elizabeth Murphy, “Planning and Exploring Under Uncertainty”, Robotics Research Group, Department of Engineering Science, University of Oxford, 2010, 244 pages. [cited by applicant]
Carbone, et al., “Augmenting situation awareness via model-based control in rescue robots”, IEEE, International Conference on Intelligent Robots and Systems, 2005, 07 pages. [cited by applicant]
Doroodgar, et al., “A hierarchical reinforcement learning based control architecture for semi-autonomous rescue robots in cluttered environments”, IEEE, International Conference on Automation Science and Engineering, 20… [cited by applicant]
Magnabosco, et al., “Cross-spectral visual simultaneous localization and mapping (SLAM) with sensor handover”, Elsevier: Robotics and Autonomous Systems, vol. 61, Issue 2, Feb. 2013, pp. 195-208. [cited by applicant]
Andrew J. Davison, “Real-time simultaneous localisation and mapping with a single camera”, Proceedings of Ninth IEEE International Conference on Computer Vision, Oct. 13, 2003, 08 pages. [cited by applicant]
International Search Report and Written Opinion of PCT Application No. PCT/JP2013/002059, issued on Jun. 28, 2013, 12 pages of ISRWO. [cited by applicant]
International Preliminary Report on Patentability of PCT Application No. PCT/JP2013/002059, issued on Nov. 6, 2014, 11 pages of IPRP. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/181,051, issued on Feb. 16, 2022, 07 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/181,051, issued on May 27, 2022, 07 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 15/816,500, issued on Dec. 12, 2019, 14 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 15/816,500, issued on Feb. 25, 2019, 13 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 15/816,500, issued on Aug. 20, 2019, 14 pages. [cited by applicant]
Advisory Office Action for U.S. Appl. No. 15/816,500, issued on Nov. 8, 2019, 03 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 15/816,500, issued on Nov. 20, 2020, 09 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 15/816,500, issued on Jun. 10, 2020, 10 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 15/384,754, issued on Apr. 6, 2017, 10 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 15/384,754, issued on Aug. 2, 2017, 07 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 15/162,246, issued on Sep. 23, 2016, 08 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/391,874, issued on Oct. 27, 2015, 09 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 14/391,874, issued on Mar. 1, 2016, 08 pages. [cited by applicant]