IP Library Granted Patent US 10,726,521
Granted Patent B2
US 10,726,521 · App. 15/973,456 · Granted Jul 28, 2020

Dynamic adaptation of device interfaces in a voice-based system

Inventor: Jian Wei Leong (Mountain View, CA)
Assignee: GOOGLE LLC
G06T3/0093G06F3/167G06K9/00369G06T3/608G06F40/205G06F40/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,726,521
App. No.
15/973,456
Granted
Jul 28, 2020
Kind
B2
Abstract

Implementations relate to dynamic adaptation of images for projection by a projector, based on one or more properties of user(s) that are in an environment with the projector. The projector can be associated with an automated assistant client of a client device. In some versions of those implementations, a pose of a user in the environment is determined and, based on the pose, a base image for projecting onto a surface is warped to generate a transformed image. The transformed image, when projected onto a surface and viewed from the pose of the user, mitigates perceived differences relative to the base image.

Claims (67)

1. A system to generate interfaces in an audio-based networked system, comprising: a computing device comprising one or more processors and a memory, the one or more processors configured to execute:

a natural language processor to:

receive an input audio signal detected by a sensor at a client device;

parse input audio signal to identify a first request in the input audio signal and a keyword associated with the first request;

a content selector component to:

select a first base digital component based on at least the first request;

select a second base digital component based on at least the keyword associated with the first request; and

a transformation component to:

determine a distance between the client device and a projection surface;

determine, based on the distance between the client device and the projection surface, transformation parameters for the first base digital component and the second base digital component, the transformation parameters configured to correct a skew of images projected onto the projection surface;

generate a first transformed image based at least on the transformation parameters and the first base digital component and a second transformed image based at least on the transformation parameters and the second base digital component; and

transmit the first transformed image and the second transformed image to the client device for projection onto the projection surface.

2. The system of claim 1 , comprising the transformation component to:

generate a first set of transformation parameters for the first base digital component; and

generate a second set of transformation parameters for the second base digital component.

3. The system of claim 1 , comprising the transformation component to:

determine a first pose of a first user; and

determine the transformation parameters based on the first pose of the first user.

4. The system of claim 3 , comprising the transformation component to:

determine a second pose of the first user; and

determine updated transformation parameters based on the second pose of the first user.

5. The system of claim 1 , comprising the transformation component to:

determine a first distance between a first user and the projection surface; and

determine the transformation parameters based on the first distance between the first user and the projection surface.

6. The system of claim 1 , wherein the transformation parameters comprise at least one of a rotation adjustment, a scale adjustment, and a skew adjustment.

7. The system of claim 1 , comprising the transformation component to:

determine, based on a first pose of the user, a predetermined size for a projection of the first transformed image; and

generate the transformation parameters based on the predetermined size for the projection of the first transformed image.

8. The system of claim 1 , comprising the content selector component to:

select the first base digital component based on a first pose of a first user.

9. The system of claim 1 , comprising the content selector component to:

select the first base digital component based on the distance between the client device and the projection surface.

10. The system of claim 1 , comprising the transformation component to:

determine a first pose of a first user is within a threshold distance of the projection surface; and

generate the first base digital component to include one or more interactive interface elements based on the first pose of the first user being within the threshold distance.

11. The system of claim 1 , comprising:

the transformation component to determine a first pose of a first user and a second pose of a second user; and

the content selector component to select the first digital component based on the first pose of the first user and the second base digital component based on the second pose of the second user.

12. A method to generate interfaces in an audio-based networked system, comprising:

receiving, by a natural language processor executed by one or more processors of a computing device, an input audio signal detected by a sensor at a client device;

parsing, by the natural language processor, the input audio signal to identify a first request in the input audio signal and a keyword associated with the first request;

selecting, by a content selector component of the computing device, a first base digital component based on at least the first request;

selecting, by the content selector component, a second base digital component based on at least the keyword associated with the first request;

determining, by a transformation component executed by the one or more processors of the computing device and based on sensor data from the client device, a distance between the client device and a projection surface;

determining, by the transformation component, based on the distance between the client device and the projection surface, transformation parameters for the first base digital component and the second base digital component, the transformation parameters configured to correct a skew of images projected onto the projection surface;

generating, by the transformation component, a first transformed image based at least on the transformation parameters and the first base digital component and a second transformed image based at least on the transformation parameters and the second base digital component; and

transmitting, by the transformation component, the first transformed image and the second transformed image to the client device for projection onto the projection surface.

13. The method of claim 12 , comprising:

generating, by the transformation component, a first set of transformation parameters for the first base digital component; and

generating, by the transformation component, a second set of transformation parameters for the second base digital component.

14. The method of claim 12 , comprising:

determining, by the transformation component, a first pose of a first user; and

determining, by the transformation component, the transformation parameters based on the first pose of the first user.

15. The method of claim 14 , comprising:

determining, by the transformation component, a second pose of the first user; and

determining, by the transformation component, updated transformation parameters based on the second pose of the first user.

16. The method of claim 12 , comprising:

determining, by the transformation component, a first distance between a first user and the projection surface; and

determining, by the transformation component, the transformation parameters based on the first distance between the first user and the projection surface.

17. The method of claim 12 , wherein the transformation parameters comprise at least one of a rotation adjustment, a scale adjustment, and a skew adjustment.

18. The method of claim 12 , further comprising:

determining, by the transformation component and based on a first pose of the user, a predetermined size for a projection of the first transformed image; and

generating, by the transformation component, the transformation parameters based on the predetermined size for the projection of the first transformed image.

19. The method of claim 12 , comprising:

determining, by the transformation component, a first pose of a first user is within a threshold distance of the projection surface; and

generating, by the transformation component, the first base digital component to include one or more interactive interface elements based on the first pose of the first user being within the threshold distance.

20. The method of claim 12 , comprising: determining, by the transformation component, a first pose of a first user and a second pose of a second user; and selecting, by the content selector component, the first base digital component based on the first pose of the first user and the second base digital component based on the second pose of the second user.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 28, 2018
From: LEONG, JIAN WEI
To: GOOGLE LLC
Reel/Frame 046722/0611 →
Continuity (2)
Continuation In Part 15955297 · Apr 17, 2018
Related Publication 20190318450A1 · Oct 17, 2019
Cited By (2)
US 12,424,222 US 12,548,558