IP Library Granted Patent US 11,848,972
Granted Patent B2
US 11,848,972 · App. 16/333,438 · Granted Dec 19, 2023

Multi-device audio streaming system with synchronization

Inventors: Richard Adams (Rolling Hills Estates, CA); Dave Hargreaves (Cambridge, GB); Elisa Moncayo (Fullerton, CA); Brock Maxwell Seiler (Jefferson Valley, NY)
Assignee: HAPTECH HOLDINGS, INC.
H04L65/60A63F13/00A63F13/235A63F13/285A63F13/327A63F13/54G06F3/011G06F3/016G06F3/017G06F3/165H04B1/385H04L65/80A63F13/215G06F2203/013G10L21/16
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,848,972
App. No.
16/333,438
Granted
Dec 19, 2023
Kind
B2
Abstract

Embodiments include an electronic control unit comprising an audio input device for receiving an audio stream from an external audio source, the audio stream being split between an audio path and a haptic path; a wireless transceiver in the haptic path for transmitting the audio stream to at least one wearable haptic device using short-range wireless communication; and a processor coupled to the transceiver and configured to calculate an amount of latency associated with transmission of the audio stream to the wearable haptic device(s), and partition the audio stream into a plurality of audio packets including a time-to-play based on the calculated latency. The control unit further includes a buffer in the audio path for inserting a time delay into the audio stream based on the calculated latency, and an audio output device in the audio path for outputting the time-delayed audio stream to an external audio listening device.

Claims (57)

1. An electronic control unit, comprising:

at least one input device for receiving audio content and data content from at least one external source;

an audio splitter configured to split the audio content between: an audio path for providing the audio content as an audio output via an external audio listening device, and a haptic path for providing the same audio content as a haptic output via a plurality of wearable haptic devices;

a wireless transceiver configured to transmit a first portion of the audio and data content to a first one of the wearable haptic devices using short-range wireless communication technology;

a processor coupled to: the haptic path for receiving the audio content therefrom, the at least one input device for receiving the data content therefrom, and the wireless transceiver for providing the first portion of the audio and data content thereto, the processor being configured to:

calculate an amount of latency due to the wireless transmission of the first portion of the audio and data content to the first one of the wearable haptic devices,

partition the audio content into a plurality of audio packets, each audio packet including a time-to-play based on the calculated amount of latency,

partition the data content into a plurality of data packets, the data content comprising control data for controlling operation of one or more of the wearable haptic devices,

interleave the data packets into the plurality of audio packets, each of the data packets and the audio packets comprising header information configured to identify one of the wearable haptic devices as a destination of the packet,

provide a first group of the interleaved audio and data packets to the wireless transceiver for transmission to the first one of the wearable haptic devices based on the destination included in the corresponding header information, the first group comprising the first portion of the audio and data content, and

provide a second group of the interleaved audio and data packets to a second one of the wearable haptic devices based on the destination included in the corresponding header information, the second group comprising a second portion of the audio and data content different from the first portion;

a buffer in communication with the processor and included in the audio path, the buffer configured to insert a time delay into the audio content provided to the audio path, the time delay being based on the calculated amount of latency; and

an audio output device coupled to the buffer for outputting the time-delayed audio content to the external audio listening device.

2. The electronic control unit of claim 1 , further comprising a low pass filter included in the haptic path for removing high frequency components from the audio content provided to the haptic path.

3. The electronic control unit of claim 1 , wherein each audio packet further includes a sync word and an identifier associated with the audio packet.

4. The electronic control unit of claim 1 , wherein the at least one input device comprises a data port coupled to the processor for receiving the data content from one or more external data sources.

5. The electronic control unit of claim 1 , wherein the wireless transceiver is further configured to receive data captured by the first one of the wearable haptic devices.

6. The electronic control unit of claim 4 , wherein the data port is a universal serial bus (USB) port.

7. The electronic control unit of claim 1 , wherein the at least one input device comprises a microphone.

8. The electronic control unit of claim 1 , wherein the wireless transceiver is a Bluetooth transceiver.

9. An entertainment system, comprising:

a plurality of wearable haptic devices, each haptic device configured to convert a corresponding portion of audio content into haptic signals; and

an electronic control unit comprising:

at least one input device for receiving the audio content and data content from at least one external source,

an audio output device for outputting the audio content to an external audio listening device,

a processor for providing the same audio content and the data content to the wearable haptic devices,

an audio splitter configured to receive the audio content from the at least one input device and provide the audio content to each of the audio output device and the processor, and

a wireless transceiver coupled to the processor and configured to transmit a first portion of the audio and data content to a first one of the wearable haptic devices using short-range wireless communication technology,

wherein the processor is configured to synchronize delivery of the haptic signals with delivery of the audio content to the audio listening device by:

calculating an amount of latency due to the wireless transmission of the first portion of the audio and data content to the first one of the wearable haptic devices,

partitioning the audio content into a plurality of audio packets, each audio packet including a time-to-play based on the calculated amount of latency,

partitioning the data content into a plurality of data packets, the data content comprising control data for controlling operation of one or more of the wearable haptic devices,

interleaving the data packets into the plurality of audio packets, each of the data packets and the audio packets comprising header information configured to identify one of the wearable haptic devices as a destination of the packet,

inserting a time delay into the audio content provided to the audio listening device, the time delay being based on the calculated amount of latency,

providing a first group of the interleaved audio and data packets to the wireless transceiver for transmission to the first one of the wearable haptic devices based on the destination included in the corresponding header information, the first group comprising the first portion of the audio and data content, and

providing a second group of the interleaved audio and data packets to a second one of the wearable haptic devices based on the destination included in the corresponding header information, the second group comprising a second portion of the audio and data content different from the first portion.

10. The entertainment system of claim 9 , wherein the at least one input device comprises a data port communicatively coupled to the processor and configured to receive the data content from an external data source.

11. The entertainment system of claim 9 , wherein each wearable haptic device is further configured to:

obtain data associated with a movement detected by the wearable haptic device, and

transmit the data to the electronic control unit.

12. The entertainment system of claim 9 , wherein the first one of the wearable haptic devices is a first footwear device comprising a first haptic transducer and the second one of the wearable haptic devices is a second footwear device comprising a second haptic transducer.

13. The entertainment system of claim 9 , wherein the first one of the wearable haptic devices includes a second wireless transceiver for receiving the first portion of the audio and data packets from the electronic control unit.

14. The entertainment system of claim 13 , wherein the wireless transceivers of the electronic control unit and the first one of the wearable haptic devices are Bluetooth transceivers.

15. The entertainment system of claim 9 , wherein each wearable haptic device includes a second processor configured to calculate a new time-to-play value for each audio packet received from the electronic control unit based on the time-to-play included in the received audio packet and a current play time provided by an internal clock included in the wearable haptic device.

16. A method of synchronizing delivery of an audio stream with delivery of haptic signals derived from the audio stream, comprising:

establishing a wireless connection with a first one of a plurality of wearable haptic devices using a wireless transceiver configured for short-range wireless communications;

receiving audio content from an external audio source via at least one input device;

providing the same audio content to each of an audio path and a haptic path, the audio path coupled to a buffer and an audio output device, and the haptic path coupled to the wireless transceiver and a processor;

receiving, at the processor, data content from an external data source via the at least one input device;

calculating, using the processor, an amount of latency due to wireless transmission of a first portion of the audio and data content by the wireless transceiver to the first one of the wearable haptic devices;

partitioning, using the processor, the audio content of the haptic path into a plurality of audio packets, each audio packet including a time-to-play based on the calculated amount of latency;

partitioning, using the processor, the data content into a plurality of data packets, the data content comprising control data for controlling operation of one or more of the wearable haptic devices,

interleaving, using the processor, the data packets into the plurality of audio packets, each of the data packets and the audio packets comprising header information configured to identify one of the wearable haptic devices as a destination of the packet,

inserting, using the buffer, a time delay into the audio stream of the audio path, the time delay being based on the calculated amount of latency;

providing the time-delayed audio stream to the audio output device for delivery to an external audio listening device coupled to the audio output device;

transmitting, via the wireless transceiver, a first group of the interleaved audio and data packets to the first one of the wearable haptic devices based on the destination included in the corresponding header information; and

providing, via the processor, a second group of the interleaved audio and data packets to a second one of the wearable haptic devices based on the destination included in the corresponding header information, the second group comprising a second portion of the audio and data content different from the first portion.

Assignments (7)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 12, 2023
From: SONICSENSORY, INC.
To: HAPTECH HOLDINGS, INC.
Reel/Frame 065222/0652 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENT NUMBER AND APPLICATION NUMBER PREVIOUSLY RECORDED AT REEL: 063500 FRAME: 0340. ASSIGNOR(S) HEREBY CONFIRMS THE LOAN FORECLOSER AND INTELLECTUAL PROPERTY ASSET TRANSFER. Recorded Sep 15, 2023
From: SONICSENSORY, INC.
To: HAPTECH HOLDINGS, INC.
Reel/Frame 064918/0160 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENT NUMBER 11306127 WAS ERRONEOUSLY INCLUDED. IT IS NOT PART OF THIS CONVEYANCE PREVIOUSLY RECORDED ON REEL 063822 FRAME 0684. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT OF ASSIGNORS INTEREST. Recorded Sep 15, 2023
From: SONICSENSORY, INC.
To: HAPTECH HOLDINGS, INC.
Reel/Frame 064918/0876 →
LOAN FORECLOSURE AND INTELLECTUAL PROPERTY ASSET TRANSFER Recorded May 1, 2023
From: SONICSENSORY, INC.
To: HAPTECH HOLDINGS, INC.
Reel/Frame 063500/0340 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 8, 2023
From: SONICSENSORY, INC.
To: HAPTECH HOLDINGS, INC.
Reel/Frame 063822/0684 →
SECURITY INTEREST Recorded Jan 22, 2021
From: SONICSENSORY, INC.
To: SILICON VALLEY BANK
Reel/Frame 054995/0553 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 14, 2019
From: ADAMS, RICHARD; HARGREAVES, DAVE; MONCAYO, ELISA; SEILER, BROCK MAXWELL
To: SONICSENSORY, INC.
Reel/Frame 048601/0673 →
Continuity (2)
Provisional Application 62394707 · Sep 14, 2016
Related Publication 20190215349A1 · Jul 11, 2019
Cited By (2)
US 12,556,147 US 12,659,050