IP Library Granted Patent US 10,037,628
Granted Patent B2
US 10,037,628 · App. 15/713,801 · Granted Jul 31, 2018

Image processing device, image processing method, and program

Inventors: Yasuhiro Suto (Tokyo, JP); Masaki Fukuchi (Tokyo, JP); Kenichirou Ooi (Kanagawa, JP); Jingjing Guo (Tokyo, JP); Kouichi Matsuda (Tokyo, JP)
Assignee: SONY CORPORATION
G06T19/006G02B27/0172G06T11/60G06T19/003G09G5/00G02B2027/014
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,037,628
App. No.
15/713,801
Granted
Jul 31, 2018
Kind
B2
Abstract

There is provided an image processing device including: a data storage unit storing feature data indicating a feature of appearance of one or more physical objects; an environment map building unit for building an environment map based on an input image obtained by imaging a real space and the feature data, the environment map representing a position of a physical object present in the real space; a control unit for acquiring procedure data for a set of procedures of operation to be performed in the real space, the procedure data defining a correspondence between a direction for each procedure and position information designating a position at which the direction is to be displayed; and a superimposing unit for generating an output image by superimposing the direction for each procedure at a position in the input image determined based on the environment map and the position information, using the procedure data.

Claims (54)

1. An apparatus comprising:

one or more processors configured to

control to display first information associated with a first procedure ID,

determine whether a first condition is satisfied based upon first conditional data associated with the first procedure ID and a captured image,

control to display second information associated with a second procedure ID upon determining that the first condition is satisfied,

determine whether a second condition is satisfied based upon second conditional data associated with the second procedure ID and a captured image, and

control to display third information associated with a third procedure ID upon determining that the second condition is satisfied,

wherein the first information, the second information, and the third information are defined to be displayed in order.

2. The apparatus according to claim 1 , wherein

the first information is displayed at a location based upon first position information associated with the first procedure ID,

the second information is displayed at a location based upon second position information associated with the second procedure ID, and

third information is displayed at a location based upon third position information associated with the third procedure ID.

3. The apparatus according to claim 2 , wherein the first position information, the second position information, and the third position information each correspond to location of one of more physical objects.

4. The apparatus according to claim 1 ,

wherein a real object in a real world is indicated by one or more selected from the group consisting of the first position information, the second position information, and the third position information, and

wherein the one or more processors is further configured to control to display, on or in vicinity of a location corresponding to the real object, one or more selected from the group consisting of the first information, the second information, and the third information.

5. The apparatus according to claim 4 , wherein the one or more selected from the group consisting of the first information, the second information, and the third information is displayed to be visually superimposed upon the real object or in the vicinity of the location corresponding to the real object.

6. The apparatus according to claim 1 , wherein one or more selected from the group consisting of the first information, the second information, and the third information includes a text description to be displayed.

7. The apparatus according to claim 1 , wherein the one or more processors is further configured to

generate a 3-D map based upon one or more captured images, and

identify a position and attitude of the apparatus based upon the one or more captured images.

8. The apparatus according to claim 1 , wherein the first conditional data represents a progress condition of a procedure associated with the first procedure ID, and the second conditional data represents a progress condition of a procedure associated with the second procedure ID.

9. The apparatus according to claim 1 , wherein the captured image based upon which the determination made as to whether the first condition is satisfied is different than the captured image based upon which the determination made as to whether the second condition is satisfied.

10. The apparatus according to claim 1 , wherein the first information, the second information, and the third information are defined to be displayed in a sequential order representing chronological steps a user performs.

11. A method comprising:

displaying first information associated with a first procedure ID;

determining whether a first condition is satisfied based upon first conditional data associated with the first procedure ID and a captured image;

displaying second information associated with a second procedure ID upon determining that the first condition is satisfied;

determining whether a second condition is satisfied based upon second conditional data associated with the second procedure ID and a captured image; and

displaying third information associated with a third procedure ID upon determining that the second condition is satisfied,

wherein the first information, the second information and the third information are defined to be displayed in order.

12. The method according to claim 1 , wherein

the first information is displayed at a location based upon first position information associated with the first procedure ID,

the second information is displayed at a location based upon second position information associated with the second procedure ID, and

third information is displayed at a location based upon third position information associated with the third procedure ID.

13. The method according to claim 12 , wherein the first position information, the second position information, and the third position information each correspond to location of one of more physical objects.

14. The method according to claim 11 , further comprising:

displaying, on or in vicinity of a location corresponding to the real object, one or more selected from the group consisting of the first information, the second information, and the third in formation,

wherein a real object in a real world is indicated by one or more selected from the group consisting of the first position information, the second position information, and the third position information.

15. The method according to claim 14 , wherein the one or more selected from the group consisting of the first information, the second information, and the third information is displayed to be visually superimposed upon the real object or in the vicinity of the location corresponding to the real object.

16. The method according to claim 11 , wherein one or more selected from the group consisting of the first information, the second information, and the third information includes a text description to be displayed.

17. The method according to claim 11 , further comprising:

generating a 3-D map based upon one or more captured images; and

identifying a position and attitude of the apparatus based upon the one or more captured images.

18. The method according to claim 11 , wherein the first conditional data represents a progress condition of a procedure associated with the first procedure ID, and the second conditional data represents a progress condition of a procedure associated with the second procedure ID.

19. The method according to claim 1 , wherein the captured image based upon which the determination made as to whether the first condition is satisfied is different than the captured image based upon which the determination made as to whether the second condition is satisfied.

20. The method according to claim 11 , wherein the first information, the second information and the third information are defined to be displayed in a sequential order representing chronological steps a user performs.

21. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising:

displaying first information associated with a first procedure ID;

determining whether a first condition is satisfied based upon first conditional data associated with the first procedure ID and a captured image;

displaying second information associated with a second procedure ID upon determining that the first condition is satisfied;

determining whether a second condition is satisfied based upon second conditional data associated with the second procedure ID and a captured image; and

displaying third information associated with a third procedure ID upon determining that the second condition is satisfied,

wherein the first information, the second information and the third information are defined to be displayed in order.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 25, 2017
From: SUTO, YASUHIRO; FUKUCHI, MASAKI; OOI, KENICHIROU; GUO, JINGJING; MATSUDA, KOUICHI
To: SONY CORPORATION
Reel/Frame 043677/0563 →
Priority Claims (1)
JP 2010-021368 · Feb 2, 2010 · national
Continuity (5)
Continuation 14991477 · Jan 8, 2016
Continuation 14869063 · Sep 29, 2015
Continuation 14527148 · Oct 29, 2014
Continuation 12984847 · Jan 5, 2011
Related Publication 20180012415A1 · Jan 11, 2018