IP Library › Granted Patent US 10,431,007
Granted Patent B2
US 10,431,007 · App. 15/609,113 · Granted Oct 1, 2019

Method and system for user interaction

Inventors: Damien Douxchamps (Kyoto, JP); Tero Aaltonen (Taipei, TW); Peter Antoniac (Oulu, FI); Harri Kovalainen (Oulu, FI)
Assignee: AUGUMENTA LTD.
G06T19/006G06F1/163G06F1/1686G06F3/002G06F3/011G06F3/0304G06F3/0425G06F3/0426G06F3/04847G06F3/04886
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,431,007
App. No.
15/609,113
Filed
May 31, 2017
Granted
Oct 1, 2019
Kind
B2
Examiner
CHOW, WING H
Art Unit
2621
USPC
345/156
Abstract

The present disclosure provides a method for user interaction. The method includes pointing a first device having a sensor, towards a panel defining a physical area having a plurality of markers rendered thereon, recognizing an interaction area defined by the plurality of markers on the panel, and rendering a plurality of graphical user interface elements on the panel in an Input/Output viewport. The plurality of markers includes at least one of an object and a shape recognizable by suitable computer vision means. Further, the Input/Output viewport includes a spatial volume between the first device and the panel.

Claims (35)

1. A method for user interaction, the method comprising:

pointing a sensor, mounted on head mounted smartglasses, towards a panel, wherein the smartglasses include an augmented reality display, and wherein the panel defines a physical area having a plurality of markers and a panel identifier;

using the sensor to recognize the plurality of markers on the panel and an interaction area defined by the plurality of markers;

using the sensor to recognize the panel identifier;

using the recognized panel identifier to select a layout for a graphical user interface comprising a plurality of graphical user interface elements; and

rendering the selected layout of the graphical user interface on the augmented reality display to appear to be floating in a spatial volume between the head mounted smartglasses and the interaction area defined by the plurality of markers; and to visually align with the panel;

wherein at least one of the plurality of markers comprises at least one of information and a hyperlink to the information, wherein the information comprises at least one of a location, product information, and user information;

wherein at least two of the plurality of markers are configured to be used as geometrical markers for defining a geometrical position and an orientation of the panel with respect to the head mounted smartglasses; and

wherein the method further comprises:

measuring a color of the interaction area and using the color measurement to affect the rendering of the identified plurality of graphical user interface elements; and

randomizing positions of the identified plurality of graphical user interface elements when rendering the identified plurality of graphical user interface elements.

2. The method of claim 1 , further comprising recognizing at least one of a hand, a finger and a predefined detectable object in the spatial volume between the head mounted smartglasses and the interaction area, wherein the at least one of the hand, the finger and the predefined detectable object is directed towards at least one of the plurality of graphical user interface elements as perceived by a user.

3. The method of claim 2 , wherein the at least one of the hand, the finger and the predefined detectable object is directed for performing at least one control operation of at least one of the head mounted smartglasses and a second device.

4. The method of claim 1 , further comprising rendering a new state of the plurality of graphical user interface elements in the spatial volume between the head mounted smartglasses and the interaction area.

5. The method of claim 4 , wherein rendering the new state further comprises performing a perspective correction with one or more parameters defined by recognizing the plurality of markers and their geometrical relationship with each other.

6. The method of claim 1 , further comprising combining a device identifier with the panel identifier for rendering a personalized content on the augmented reality display.

7. The method of claim 1 , further comprising detecting movement of the head mounted smartglasses, and employing the detected movement to adjust a location of the interaction area.

8. The method of claim 1 , wherein the plurality of markers and panel identifier are printed on the panel.

9. A system for user interaction, comprising:

a server comprising a database configured to store a plurality of device identifiers, a plurality of panel identifiers, and rendering information;

head mounted smartglasses having a sensor, wherein the smartglasses include an augmented reality display, and wherein the sensor is configured to be used for pointing towards a panel defining a physical area having a plurality of markers and a panel identifier rendered thereon, wherein the head mounted smartglasses are configured to:

recognize the plurality of markers on the panel and an interaction area defined by the plurality of markers;

recognize the panel identifier;

use the recognized panel identifier to select a layout for a graphical user interface comprising a plurality of graphical user interface elements; and

render the selected layout of the graphical user interface on the augmented reality display to appear to be floating in a spatial volume between the head mounted smartglasses and the interaction area defined by the plurality of markers; and to visually align with the panel;

wherein at least one of the plurality of markers comprises at least one of information and a hyperlink to the information, wherein the information comprises at least one of a location, product information, and user information,

wherein at least two of the plurality of markers are configured to be used as geometrical markers for defining a geometrical position and an orientation of the panel with respect to the head mounted smartglasses; and

wherein the system is configured to:

measure a color of the interaction area and using the color measurement to affect the rendering of the identified plurality of graphical user interface elements; and

randomize positions of the identified plurality of graphical user interface elements when rendering the identified plurality of graphical user interface elements.

10. The system of claim 9 , wherein the head mounted smartglasses are further configured to recognize at least one of a hand, a finger and a predefined detectable object in the spatial volume between the head mounted smartglasses and the interaction area, wherein the at least one of the hand, the finger and the predefined detectable object is directed towards at least one of the plurality of graphical user interface elements.

11. The system of claim 9 , wherein the head mounted smartglasses are further configured to render a new state of the plurality of graphical user interface elements in the spatial volume between the head mounted smartglasses and the interaction area.

12. system of claim 11 , wherein the head mounted smartglasses are further configured to perform a perspective correction with one or more parameters defined by recognizing the plurality of markers and their geometrical relationship with each other when rendering the plurality of graphical user interface elements.

13. The system of claim 9 , wherein the head mounted smartglasses are further configured to combine a device identifier with the panel identifier for rendering a personalized content on the augmented reality display.

14. The system of claim 9 , wherein the head mounted smartglasses further comprise at least one motion sensor configured to detect movement of the head mounted smartglasses, and wherein the head mounted smartglasses employ the detected movement to adjust a location of the interaction area.

Assignments (2)
ASSIGNMENT BY DECLARATION TO CLARIFY OWNERSHIP OF AN PATENTS. Recorded Mar 20, 2025
From: AUGUMENTA HOLDINGS OÜ
To: AUGUMENTA HOLDINGS OÜ
Reel/Frame 070709/0286 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 1, 2017
From: DOUXCHAMPS, DAMIEN; AALTONEN, TERO; ANTONIAC, PETER; KOVALAINEN, HARRI
To: AUGUMENTA LTD.
Reel/Frame 042554/0368 →
Continuity (2)
Provisional Application 62343228 · May 31, 2016
Related Publication 20170344124A1 · Nov 30, 2017