Multimedia system and method for AR/VR smart contact lens

1 . A smart contact lens (SCL) based multimedia system comprising:

a contact lens substrate comprising an embedded display component configured to present visual content to a user;

an audio output device configured to emit audio content;

at least one communication unit comprising a data receiver or transceiver configured to receive or send data wirelessly to and from an external communication unit; and

a multimedia processor operatively connected to the embedded display component and the audio output device,

wherein the multimedia processor is configured to receive a multimedia stream comprising at least video and audio content, to demultiplex the multimedia stream into separate video and audio streams, and to direct the video stream to the embedded display component and the audio stream to the audio output device in temporal synchronization.

2 . The smart contact lens system of claim 1 , wherein the multimedia processor is configured to receive the multimedia stream from an external source over a wireless or wired communication interface, or to retrieve the multimedia stream from a memory device integrated into the multimedia processor.

3 . The smart contact lens system of claim 1 , wherein the multimedia processor is either (i) onboard and integrated into the contact lens substrate, or (ii) offboard as an external component operatively connected to the smart contact lens system.

4 . The smart contact lens system of claim 1 , wherein the at least one communication unit is configured to transmit each demultiplexed data stream to a corresponding data-type-specific receiver or transceiver of a paired external communication unit.

5 . The smart contact lens system of claim 1 , wherein the multimedia processor is further configured to add text data to the video stream,

wherein the text data is generated in real time by analyzing at least one of the demultiplexed audio or video streams, or is received as part of the multimedia source data stream.

6 . A smart contact lens (SCL) based multimedia system comprising:

a contact lens substrate;

a multimedia processing device operatively connected to the smart contact lens and configured to receive and process multimedia data, wherein the multimedia processing device combines one or more of an audio stream, an image stream, a video stream, and a text stream to form a mixed multimedia dataset;

an image capture device or a video capture device positioned to acquire visual data from a forward-facing field of view and configured to transmit said visual data to the multimedia processing device;

a communication interface configured to exchange multimedia data with an external communication unit; and

the external communication unit comprising a multimedia transceiver configured to transmit or receive image or video data.

7 . The smart contact lens system of claim 6 , further comprising:

an audio capture device, external to or embedded within the contact lens substrate, wherein the audio capture device is configured to capture one or more audio channels and to transmit the captured audio to the multimedia processing device for at least one of recording, retransmission, or processing into a multimedia data stream.

8 . Smart contact lens system of claim 6 , further comprising:

external multimedia transceiver or transmitter of audio data.

9 . A method of operating a smart contact lens-based multimedia system, the method comprising:

presenting visual content to a user via a display component embedded in a contact lens substrate;

emitting audio content via an audio output device;

receiving or sending data wirelessly to and from an external communication unit via at least one communication unit comprising a data receiver or transceiver; and

receiving, by a multimedia processor operatively connected to the embedded display component and the audio output device, a multimedia stream comprising at least video and audio content;

demultiplexing, by the multimedia processor, the multimedia stream into separate video and audio streams; and

directing the video stream to the embedded display component and the audio stream to the audio output device in temporal synchronization.

10 . The method of claim 9 , further comprising:

generating, in real time, a text data stream by analyzing at least one of the demultiplexed audio stream or the demultiplexed video stream using a processing module; and

overlaying the generated text data onto the video stream presented on the embedded display component of the contact lens substrate in temporal synchronization with corresponding audio or visual content.

11 . The method of claim 9 , further comprising:

pre-processing, after the demultiplexing, one or more of the audio, video, or text streams; and

transmitting the pre-processed stream or streams to one or more of an audio receiving device, an image or video receiving device, or a text receiving device, wherein outputting of the audio stream is performed by the audio receiving device and outputting of the image or video stream to the embedded display component of the contact lens substrate is performed by the image or video receiving device.

12 . The method of claim 11 , wherein said transmitting is performed wirelessly or via wire or via light beam to the corresponding receiving device.

13 . The method of claim 11 , wherein the audio receiving device performs post-processing of the audio stream prior to outputting the audio stream, and wherein the image or video receiving device performs post-processing of the video stream prior to outputting the video stream.

14 . A method for operating a smart contact lens-based multimedia system, the method comprising:

capturing, via an audio sensor, an audio stream;

capturing, via an image or video capture sensor integrated into the smart contact lens, one or more of an image stream or a video stream;

combining one or more of the audio stream, the image stream, the video stream, and a text stream to form a mixed multimedia dataset; and

storing the mixed multimedia dataset in memory or transmitting the mixed multimedia dataset to an external communication unit.

15 . The method of claim 14 , further comprising:

pre-processing, prior to the combining, one or more of the audio stream, image stream, video stream, or text stream.

16 . A non-transitory machine-readable medium storing one or more sequences of instructions which, when executed by one or more multimedia processors operatively connected to a smart contact lens-based multimedia system, cause the system to:

receive, from an external communication unit via a wireless interface, a multimedia data stream comprising at least video data and audio data;

demultiplex the multimedia data stream into separate data-type-specific streams including a video stream and an audio stream; and

route the video stream to a display component embedded in a contact lens substrate, and route the audio stream to an audio output device in temporal synchronization.

17 . The non-transitory machine-readable medium of claim 16 , wherein the one or more-sequences of instructions further cause the smart contact lens-based multimedia system to:

receive, from an external computing unit, a real-time text stream, or demultiplex multimedia data to retrieve a text stream; and

overlay the received or retrieved text onto the video stream presented on the embedded display component of a contact lens substrate in semantic and temporal synchronization.

18 . The non-transitory machine-readable medium of claim 16 , wherein the one or more sequences of instructions further cause the smart contact lens-based multimedia system to:

pre-process, after the demultiplexing, one or more of the streams obtained from the multimedia data stream; and

transmit the pre-processed stream or streams to one or more of an audio receiving device, an image or video receiving device, or a text receiving device, wherein the audio receiving device outputs the audio stream and the image or video receiving device outputs the image or video stream to the embedded display component of the contact lens substrate.

19 . The non-transitory machine-readable medium of claim 16 , further comprising one or more instructions for:

capturing, via an audio sensor, audio data;

capturing, via an image or video capture sensor integrated into said smart contact lens, one or more of image data, video data and text data;

combining one or more of said audio data, image data, video data and text data to form a mixed media set;

storing said mixed media set in a memory;

and transmitting said mixed media set to an external communication unit.

20 . The non-transitory machine-readable medium of claim 19 , further comprising one or more instructions for pre-processing, prior to said combining, one or more of said audio data, image data, video data and text data.

21 . A computer-implemented method executed by one or more multimedia processors operatively connected to a smart contact lens-based multimedia system, the method comprising:

receiving, from an external communication unit via a wireless interface, a multimedia data stream comprising at least two types selected from video data, audio data, and text data;

demultiplexing the multimedia data stream into separate data-type-specific streams including a video stream and an audio stream; and

routing the video stream to a display component embedded in a contact lens substrate, and routing the audio stream to an audio output device in temporal synchronization.

22 . The method of claim 21 , further comprising:

receiving, from an external computing unit, a real-time text stream, or demultiplexing multimedia data to retrieve a text stream; and

overlaying the received or retrieved text onto the video stream presented on the embedded display component of the contact lens substrate in semantic and temporal synchronization.