IP Library › Granted Patent US 11,769,500
Granted Patent B2
US 11,769,500 · App. 17/225,563 · Granted Sep 26, 2023

Augmented reality-based translation of speech in association with travel

Inventors: Virginia Drummond (Venice, CA); Ilteris Kaan Canberk (Marina Del Rey, CA); Jean Luo (Los Angeles, CA); Alek Matthiessen (Marina Del Rey, CA); Celia Nicole Mourkogiannis (Los Angeles, CA)
Assignee: Snap Inc.
G10L15/22G06T19/006G10L15/26
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,769,500
App. No.
17/225,563
Granted
Sep 26, 2023
Kind
B2
Abstract

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for providing augmented reality content in association with travel. The program and method provide for receiving, by a messaging application, a request to perform a scan operation in association with an image captured by a device camera; determining a travel parameter associated with the request and an attribute of an object depicted in the image; selecting an augmented reality content item based on at least one of the travel parameter or the attribute, the augmented reality content item being configured to present augmented reality content based on speech input; receiving the speech input; obtaining at least one of a transcription or translation of the speech input; and presenting the augmented reality content item, including the transcription or translation, in association with the image.

Claims (44)

1. A method, comprising:

receiving, by a messaging application running on a device of a user, a request to perform a scan operation in association with live video captured by a camera of the device;

determining, in response to receiving the request, a travel parameter associated with the request and an attribute of an object depicted in the live video;

selecting, from plural augmented reality content items, an augmented reality content item based on at least one of the travel parameter or the attribute, the augmented reality content item being configured to present respective augmented reality content based on speech input associated with the request;

activating the selected augmented reality content item;

receiving the speech input in association with activating the selected augmented reality content item;

obtaining at least one of a transcription or a translation of the speech input; and

presenting the selected augmented reality content item in association with the live video, the selected augmented reality content item causing the live video to be modified with the respective augmented reality content and to include the at least one of the transcription or the translation.

2. The method of claim 1 , wherein the travel parameter indicates at least one of a travel schedule, a transportation schedule, a language, a general location, a specific venue or landmark, an activity, a list of participants, or a topic of interest associated with travel by the user.

3. The method of claim 1 , wherein the attribute indicates that the object corresponds to an individual other than the user in the live video image, the individual providing the speech input corresponding to live speech.

4. The method of claim 3 , wherein the augmented reality content item is selected so as to provide at least one of a visual translation of the live speech or an audio translation of the live speech.

5. The method of claim 1 , wherein the attribute indicates that the object corresponds to a video being displayed on a second device, the video being displayed with audio output corresponding to the speech input.

6. The method of claim 5 , wherein the augmented reality content item is selected so as to provide at least one of a visual transcription of the audio output, a visual translation of the audio output, or an audio translation of the audio output.

7. The method of claim 1 , further comprising:

detecting a voice command for translating the speech input, the voice command and the speech input being provided by the user in association with the request, wherein the attribute indicates that the object corresponds to the user.

8. The method of claim 7 , wherein the augmented reality content item is selected so as to provide a visual transcription of the speech input, a visual translation of the speech input, and an audio translation of the speech input.

9. The method of claim 1 , wherein selecting the augmented reality content item is further based on a geolocation of the device.

10. A device, comprising:

a processor; and

a memory storing instructions that, when executed by the processor, configure the processor to:

receive, by a messaging application running on a device of a user, a request to perform a scan operation in association with live video captured in real-time by a camera of the device;

determine, in response to receiving the request, a travel parameter associated with the request and an attribute of an object depicted in the live video;

select, from plural augmented reality content items, an augmented reality content item based on at least one of the travel parameter or the attribute, the augmented reality content item being configured to present respective augmented reality content based on speech input associated with the request;

activate the selected augmented reality content item;

receive the speech input in association with activating the selected augmented reality content item;

obtain at least one of a transcription or a translation of the speech input; and

present the selected augmented reality content item in association with the live video, the selected augmented reality content item causing the live video to be modified with the respective augmented reality content and to include the at least one of the transcription or the translation.

11. The device of claim 10 , wherein the travel parameter indicates at least one of a travel schedule, a transportation schedule, a language, a general location, a specific venue or landmark, an activity, a list of participants, or a topic of interest associated with travel by the user.

12. The device of claim 10 , wherein the attribute indicates that the object corresponds to an individual other than the user in the live video, the individual provide the speech input corresponding to live speech.

13. The device of claim 12 , wherein the augmented reality content item is selected so as to provide at least one of a visual translation of the live speech or an audio translation of the live speech.

14. The device of claim 10 , wherein the attribute indicates that the object corresponds to a video being displayed on a second device, the video being displayed with audio output corresponding to the speech input.

15. The device of claim 14 , wherein the augmented reality content item is selected so as to provide at least one of a visual transcription of the audio output, a visual translation of the audio output, or an audio translation of the audio output.

16. The device of claim 10 , wherein the instructions further configure the processor to:

detect a voice command for translating the speech input, the voice command and the speech input being provided by the user in association with the request, wherein the attribute indicates that the object corresponds to the user.

17. The device of claim 16 , wherein the augmented reality content item is selected so as to provide a visual transcription of the speech input, a visual translation of the speech input, and an audio translation of the speech input.

18. The device of claim 10 , wherein selecting the augmented reality content item is further based on a geolocation of the device.

19. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

receive, by a messaging application running on a device of a user, a request to perform a scan operation in association with live video captured in real-time by a camera of the device;

determine, in response to receiving the request, a travel parameter associated with the request and an attribute of an object depicted in the live video;

select, from plural augmented reality content items, an augmented reality content item based on at least one of the travel parameter or the attribute, the augmented reality content item being configured to present respective augmented reality content based on speech input associated with the request;

activate the selected augmented reality content item;

receive the speech input in association with activating the selected augmented reality content item;

obtain at least one of a transcription or a translation of the speech input; and

present the selected augmented reality content item in association with the live video, the selected augmented reality content item causing the live video to be modified with the respective augmented reality content and to include the at least one of the transcription or the translation.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 11, 2023
From: DRUMMOND, VIRGINIA; KAAN CANBERK, ILTERIS; LUO, JEAN; MATTHIESSEN, ALEK; MOURKOGIANNIS, CELIA NICOLE
To: SNAP INC.
Reel/Frame 064564/0111 →
Continuity (2)
Provisional Application 63046114 · Jun 30, 2020
Related Publication 20210407506A1 · Dec 30, 2021