IP Library Granted Patent US 11,581,007
Granted Patent B2
US 11,581,007 · App. 17/302,191 · Granted Feb 14, 2023

Preventing audio delay-induced miscommunication in audio/video conferences

Inventors: Tiberiu Suto (Franklin, NY); Shikhar Kwatra (San Jose, CA); Hemant Kumar Sivaswamy (Pune, IN); Clement Decrop (Arlington, VA)
Assignee: Kyndryl, Inc.
G10L21/10G06N3/0454G10L15/16G10L15/1815G10L15/22G10L15/30H04L7/0008H04N7/15
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,581,007
App. No.
17/302,191
Granted
Feb 14, 2023
Kind
B2
Abstract

Embodiments for delay-induced miscommunication reduction are provided. The embodiment may include capturing data streams transmitted between participants in an A/V exchange; translating, on a sender device prior to transmission to a recipient device, an audio stream within the data streams to text; timestamping, on a sender device prior to transmission to the recipient device, each word in the translated audio stream; transmitting the audio stream and the sender-side translated and timestamped audio stream to the recipient device; translating, on the recipient device, the transmitted audio stream to text; timestamping, on the recipient device, each word in the translated audio stream; determining a lag exists in the A/V exchange based on a comparison of each timestamp for corresponding words on the sender-side translated and timestamped audio stream and the recipient-side translated and timestamped audio stream; and generating a true transcript of an intended exchange between the participants based on the comparison.

Claims (55)

1. A processor-implemented method, the method comprising:

capturing one or more data streams transmitted between participants in an audio/video (A/V) exchange;

translating, on a sender device prior to transmission to a recipient device, an audio stream within the one or more data streams to text;

timestamping, on a sender device prior to transmission to the recipient device, each word in the translated audio stream;

transmitting the audio stream and the sender-side translated and timestamped audio stream to the recipient device;

translating, on the recipient device, the transmitted audio stream to text;

timestamping, on the recipient device, each word in the translated audio stream;

determining a lag exists in the A/V exchange based on a comparison of each timestamp for corresponding words on the sender-side translated and timestamped audio stream and the recipient-side translated and timestamped audio stream; and

generating a true transcript of an intended exchange between the participants based on the comparison.

2. The method of claim 1 , further comprising:

identifying one or more points in the translated audio stream where the determined lag satisfies a preconfigured lag threshold; and

determining an intended order of communication for the one or more identified points based on natural language processing and the comparison.

3. The method of claim 1 , wherein generation of the true transcript is performed in real-time and reorganizes words or phrases based on the determined lag in the comparison to align with a speaker intention.

4. The method of claim 1 , further comprising:

displaying the generated true transcript, in real-time, on a graphical user interface associated with a software program enabling the A/V exchange.

5. The method of claim 1 , wherein generated true transcript is an audio reproduction of the audio stream.

6. The method of claim 5 , wherein the audio reproduction is generated using a Generated Adversarial Network (GAN) to fill any delay gaps in the audio stream.

7. The method of claim 1 , wherein the sender device and the recipient device perform time synchronization using a common high-resolution source and a high-resolution signal, wherein the common high-resolution source is selected from a group consisting of an atomic clock or a common Network Time Protocol (NTP).

8. A computer system, the computer system comprising:

one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising:

capturing one or more data streams transmitted between participants in an audio/video (A/V) exchange;

translating, on a sender device prior to transmission to a recipient device, an audio stream within the one or more data streams to text;

timestamping, on a sender device prior to transmission to the recipient device, each word in the translated audio stream;

transmitting the audio stream and the sender-side translated and timestamped audio stream to the recipient device;

translating, on the recipient device, the transmitted audio stream to text;

timestamping, on the recipient device, each word in the translated audio stream;

determining a lag exists in the A/V exchange based on a comparison of each timestamp for corresponding words on the sender-side translated and timestamped audio stream and the recipient-side translated and timestamped audio stream; and

generating a true transcript of an intended exchange between the participants based on the comparison.

9. The computer system of claim 8 , further comprising:

identifying one or more points in the translated audio stream where the determined lag satisfies a preconfigured lag threshold; and

determining an intended order of communication for the one or more identified points based on natural language processing and the comparison.

10. The computer system of claim 8 , wherein generation of the true transcript is performed in real-time and reorganizes words or phrases based on the determined lag in the comparison to align with a speaker intention.

11. The computer system of claim 8 , further comprising:

displaying the generated true transcript, in real-time, on a graphical user interface associated with a software program enabling the A/V exchange.

12. The computer system of claim 8 , wherein generated true transcript is an audio reproduction of the audio stream.

13. The computer system of claim 12 , wherein the audio reproduction is generated using a Generated Adversarial Network (GAN) to fill any delay gaps in the audio stream.

14. The computer system of claim 8 , wherein the sender device and the recipient device perform time synchronization using a common high-resolution source and a high-resolution signal, wherein the common high-resolution source is selected from a group consisting of an atomic clock or a common Network Time Protocol (NTP).

15. A computer program product, the computer program product comprising:

one or more computer-readable tangible storage medium and program instructions stored on at least one of the one or more tangible storage medium, the program instructions executable by a processor capable of performing a method, the method comprising:

capturing one or more data streams transmitted between participants in an audio/video (A/V) exchange;

translating, on a sender device prior to transmission to a recipient device, an audio stream within the one or more data streams to text;

timestamping, on a sender device prior to transmission to the recipient device, each word in the translated audio stream;

transmitting the audio stream and the sender-side translated and timestamped audio stream to the recipient device;

translating, on the recipient device, the transmitted audio stream to text;

timestamping, on the recipient device, each word in the translated audio stream;

determining a lag exists in the A/V exchange based on a comparison of each timestamp for corresponding words on the sender-side translated and timestamped audio stream and the recipient-side translated and timestamped audio stream; and

generating a true transcript of an intended exchange between the participants based on the comparison.

16. The computer program product of claim 15 , further comprising:

identifying one or more points in the translated audio stream where the determined lag satisfies a preconfigured lag threshold; and

determining an intended order of communication for the one or more identified points based on natural language processing and the comparison.

17. The computer program product of claim 15 , wherein generation of the true transcript is performed in real-time and reorganizes words or phrases based on the determined lag in the comparison to align with a speaker intention.

18. The computer program product of claim 15 , further comprising:

displaying the generated true transcript, in real-time, on a graphical user interface associated with a software program enabling the A/V exchange.

19. The computer program product of claim 15 , wherein generated true transcript is an audio reproduction of the audio stream.

20. The computer program product of claim 19 , wherein the audio reproduction is generated using a Generated Adversarial Network (GAN) to fill any delay gaps in the audio stream.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 18, 2021
From: INTERNATIONAL BUSINESS MACHINES CORPORATION
To: KYNDRYL, INC.
Reel/Frame 058213/0912 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 27, 2021
From: SUTO, TIBERIU; KWATRA, SHIKHAR; SIVASWAMY, HEMANT KUMAR; DECROP, CLEMENT
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 056050/0621 →