IP Library Granted Patent US 11,848,026
Granted Patent B2
US 11,848,026 · App. 17/085,082 · Granted Dec 19, 2023

Performing artificial intelligence sign language translation services in a video relay service environment

Inventor: Conrad A. Maxwell (Herriman, UT)
Assignee: Sorenson IP Holdings, LLC
G10L21/10G06T13/205G06T13/40G06V40/28G10L13/027H04N7/147H04N7/15G09B21/009G10L2021/065
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,848,026
App. No.
17/085,082
Granted
Dec 19, 2023
Kind
B2
Abstract

Video relay services, communication systems, non-transitory machine-readable storage media, and methods are disclosed herein. A video relay service may include at least one server configured to receive a video stream including sign language content from a video communication device during a real-time communication session. The server may also be configured to automatically translate the sign language content into a verbal language translation during the real-time communication session without assistance of a human sign language interpreter. Further, the server may be configured to transmit the verbal language translation during the real-time communication session.

Claims (27)

1. A video phone system for providing automated translation services during a real-time video phone communication session between two or more people, comprising:

a video communication device associated with a near-end user;

a far-end communication device associated with a far-end user;

a video relay service configured to connect to the near-end device over a first connection and to the far-end device over a second connection, thereby facilitating a communication session between the near-end user and the far-end user;

at least one server in operable communication with the video relay service, the at least one server comprising an AI translation engine, the server configured to:

receive an audio stream from one or more of the video communication device and the far-end communication device during the real-time communication session between the near-end user and the far-end user;

analyze the audio stream;

automatically translate voice content from the audio stream into sign language content during the real-time video phone communication session without assistance from a human sign language interpreter; and,

transmit the sign language content to at least the near-end user during the real-time video phone communication session.

2. The video phone system of claim 1 , wherein the server is further configured to recognize a spoken language within the audio stream and automatically translate voice content from the audio stream into sign language content corresponding to the recognized spoken language.

3. The video phone system of claim 1 , wherein the at least one server is further configured to automatically translate voice content from the audio stream into sign language content text using a speech-to-text voice recognition system to convert voice content into text.

4. The video phone system of claim 3 , wherein the at least one server is configured to transmit the text in a written language understandable by the at least one call recipient using a call recipient profile setting accessible by the at least one server.

5. The video phone system of claim 3 , wherein the at least one server is configured to transmit the sign language content in simulated avatar format depicting the-sign language corresponding to a sign language understandable by the at least one call recipient by using a call recipient profile setting accessible by the at least one server, without assistance of a human sign language interpreter.

6. The video phone system of claim 5 , wherein the at least one server is configured to transmit both the generated text and a simulated avatar format.

7. A method of performing automated translation services during a real-time communication session between two or more people using a video relay service in operable communication with server comprising an artificial intelligence (AI) translation engine, the method comprising:

establishing a first connection between a communication device of one or more call initiators and the video relay service and a second connection between the video relay service and a communication device of one or more call recipients;

using the AI translation engine to automatically translate an audio stream received from the call initiator over the first connection into sign language content converting the audio stream into text in the language of one or more of the call recipients and comparing the text to stored video files in an AI translation database; and

transmitting the text to the video communication device over the second connection during the real-time communication session.

8. The method of claim 7 , further comprising using the AI translation engine to convert the text into a simulated avatar format and transmitting the simulated avatar format to the video communication device of one or more call recipients over the second connection.

9. The method of claim 8 , further comprising transmitting the text and the simulated avatar format to the video communication device of one or more call recipients over the second connection.

10. A non-transitory machine-readable storage medium comprising processor executable instructions that, when executed by a processor of a server, cause the server to perform operations comprising:

establishing a first connection between a communication device of one or more call initiators and a video relay service in operable communication with and artificial intelligence (AI) translation engine and a second connection between the video relay service and a communication device of one or more call recipients;

receiving an audio stream from the one or more call initiators over the first connection during a real-time video phone communication session;

automatically translating the audio stream into sign language content using the AI translation engine without assistance of a human sign language interpreter by converting the audio stream into text in the language of one or more call recipients; and

transmitting the text to the video communication device of one or more call recipients over the second connection during the real-time video phone communication session.

11. The non-transitory machine-readable storage medium of claim 10 , the operations further comprising converting the text into a simulated avatar format and transmitting the simulated avatar format to the video communication device of one or more call recipients over the second connection.

12. The non-transitory machine-readable storage medium of claim 11 , the operations further comprising transmitting the text and the simulated avatar format to the video communication device of one or more call recipients over the second connection.

Assignments (5)
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA THE NAME OF THE LAST RECEIVING PARTY SHOULD BE CAPTIONCALL, LLC PREVIOUSLY RECORDED ON REEL 67190 FRAME 517. ASSIGNOR(S) HEREBY CONFIRMS THE RELEASE OF SECURITY INTEREST. Recorded May 31, 2024
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
To: SORENSON IP HOLDINGS, LLC; SORENSON COMMUNICATIONS, LLC; CAPTIONCALL, LLC
Reel/Frame 067591/0675 →
RELEASE OF SECURITY INTEREST Recorded Apr 23, 2024
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
To: SORENSON IP HOLDINGS, LLC; SORENSON COMMUNICATIONS, LLC; CAPTIONALCALL, LLC
Reel/Frame 067190/0517 →
SECURITY INTEREST Recorded Apr 23, 2024
From: SORENSON COMMUNICATIONS, LLC; INTERACTIVECARE, LLC; CAPTIONCALL, LLC
To: OAKTREE FUND ADMINISTRATION, LLC, AS COLLATERAL AGENT
Reel/Frame 067573/0201 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 7, 2023
From: MAXWELL, CONRAD A.
To: SORENSON IP HOLDINGS LLC
Reel/Frame 065486/0100 →
JOINDER NO. 1 TO THE FIRST LIEN PATENT SECURITY AGREEMENT Recorded Apr 22, 2021
From: SORENSON IP HOLDINGS, LLC
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
Reel/Frame 056019/0204 →