IP Library Granted Patent US 10,281,981
Granted Patent B2
US 10,281,981 · App. 15/415,817 · Granted May 7, 2019

AR output method and electronic device for supporting the same

Inventors: Kyung Tae Kim (Gyeonggi-do, KR); Su Young Park (Gyeonggi-do, KR); Sang Chul Baek (Seoul, KR); Jae Yung Yeo (Gyeonggi-do, KR); Young Keun Choi (Gyeonggi-do, KR)
Assignee: Samsung Electronics Co., Ltd.
G06F3/013G06F3/017G06T19/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,281,981
App. No.
15/415,817
Granted
May 7, 2019
Kind
B2
Abstract

An augmented reality (AR) output method includes generating line of sight information of a user of the electronic device based on at least one of a sensor module or a camera module of the electronic device, extracting line of sight space information corresponding to the line of sight information of the user from three-dimensional (3D) space information corresponding to a real object around the user based on the line of sight information, recognizing a user input using the camera module or the sensor module, comparing a first portion of the line of sight space information with the user input, generating a compensated input information by compensating the user input based on comparing the first portion with the user input, generating a first virtual object based on the compensated input information, and outputting the first virtual object on a display of the electronic device.

Claims (61)

1. An augmented reality (AR) output method performed in an electronic device, the method comprising:

generating line of sight information of a user of the electronic device based on at least one of a sensor module or a camera module of the electronic device;

extracting three-dimensional (3D) space information corresponding to a real object around the user based on the line of sight information;

recognizing a user input using the camera module or the sensor module;

comparing a first portion of the 3D space information with the user input;

generating compensated input information by compensating the user input based on comparing the first portion with the user input;

generating a first virtual object based on the compensated input information; and

outputting the first virtual object on a display of the electronic device,

wherein generating the first virtual object comprises:

matching each point of the user input to a corresponding point of the 3D space information based on a surface or contour of the real object, and

wherein the corresponding point is placed on an extension line connecting a line of sight with the user input.

2. The method of claim 1 , wherein the extracting of the 3D space information comprises:

generating the 3D space information based on the sensor module or the camera module.

3. The method of claim 2 , wherein the generating of the 3D space information comprises:

capturing surroundings of the electronic device using the camera module comprising at least one of a color camera, a depth camera, a thermal camera, or an infrared (IR) camera.

4. The method of claim 2 , wherein the generating of the 3D space information comprises:

generating the 3D space information comprising a line, a polyline, a curve, a polygon, a circle, an oval, and text information.

5. The method of claim 1 , wherein the recognizing of the user input comprises:

recognizing a gesture or an input pattern of the user as the user input.

6. The method of claim 1 , wherein the recognizing of the user input comprises:

recognizing the user input by one of a point input or a stroke input.

7. The method of claim 1 , wherein the comparing of the first portion and the user input comprises:

extracting a second portion in the 3D space information included in a range corresponding to a path of the user input from the 3D space information; and

measuring a distance between the second portion and a path of the user input.

8. The method of claim 1 , wherein the comparing of the first portion and the user input comprises:

calculating plane information comprising a path of the user input;

extracting a second portion in the 3D space information included in a range corresponding to a path of the user input from the 3D space information; and

comparing plane information of the second portion with plane information comprising the path of the user input.

9. The method of claim 1 , wherein the comparing of the first portion and the user input comprises:

verifying a corresponding point where the real object is met or adjacent to the extension line for connecting a location of the line of sight of the user with each of points which configure the user input.

10. The method of claim 9 , wherein the verifying of the corresponding point comprises:

determining a point, where a surface or contour of the real object is adjacent to the extension line, as the corresponding point.

11. The method of claim 1 , wherein the generating of the first virtual object comprises:

generating the first virtual object, if it is determined that the first portion in the 3D space information is similar to the user input based on similarity between the first portion and the user input.

12. The method of claim 1 , wherein the generating of the first virtual object comprises:

arranging the virtual object relative to an axis or surface parallel to a central axis or a central surface of the real object.

13. The method of claim 1 , wherein the generating of the first virtual object comprises:

generating a virtual object which is extended from a boundary of the real object and forms an angle with the boundary of the real object.

14. The method of claim 1 , wherein the recognizing of the user input comprises:

recognizing the user input of connecting a first object with a second object, and

wherein the generating of the first virtual object comprises:

generating the first virtual object connecting a connection point of the first object with a connection point of the second object using a line according to attributes.

15. The method of claim 14 , wherein the generating of the first virtual object comprises:

changing attributes of the first virtual object based on a change of the attributes, if it is determined that the attributes of the first object are changed.

16. A wearable electronic device, comprising:

a display;

a memory;

a camera module;

a sensor module; and

a processor configured to:

generate line of sight information of a user of the wearable electronic device based on at least one of the sensor module or the camera module,

extract three-dimensional (3D) space information corresponding to a real object around the user based on the line of sight information,

recognize a user input using the camera module or the sensor module,

compare a first portion of the 3D space information with the user input,

generate compensated input information by compensating the user input based on comparing the first portion with the user input,

generate a first virtual object based on the compensated input information,

match each point of the user input to a corresponding point of a 3D virtual space based on a surface or contour of the real object, and

output the first virtual object on the display,

wherein the corresponding point is placed on an extension line connecting a line of sight with the user input in the 3D virtual space.

17. The electronic device of claim 16 , wherein the wearable electronic device is one of smart glasses or a head mount device (HMD).

18. The electronic device of claim 16 , wherein the memory is configured to store information about the real object or the first virtual object included in the 3D virtual space.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 25, 2017
From: KIM, KYUNG TAE; PARK, SU YOUNG; BAEK, SANG CHUL; YEO, JAE YUNG; CHOI, YOUNG KEUN
To: SAMSUNG ELECTRONICS CO., LTD
Reel/Frame 041085/0010 →
Priority Claims (1)
KR 10-2016-0008812 · Jan 25, 2016 · national
Continuity (1)
Related Publication 20170212585A1 · Jul 27, 2017
Cited By (2)
US 12,518,307 US 12,626,470