System and method for privacy-preserving audio-visual processing
The present invention describes a privacy-preserving audio-visual (AV) processing system. The system is configured to transform raw AV data into non-reconstructable embeddings in real-time, immediately discarding the original sensitive data within milliseconds. This process enables AI-based event analysis solely on these secure embeddings, significantly mitigating privacy risks associated with traditional surveillance. Furthermore, the system incorporates a cryptographically auditable override mechanism, allowing authorized temporary access to buffered raw data, with all such events immutably logged, often via blockchain technology, ensuring accountability and regulatory compliance. The innovation's value is emphasized through its unique combination of these existing technologies, making it particularly suitable for sensitive environments like healthcare facilities and private homes where strict privacy is paramount.
1 . A method for privacy-preserving processing of audio and video data, the method comprising:
capturing, via at least one sensor, a stream of raw audio and video data, wherein the sensor is connected to a local computing device;
processing the stream of raw audio and video data using an on-device embedding generator to generate one or more non-reversible, task-specific multimodal embeddings,
wherein the one or more embeddings
comprises pattern-recognition feature vectors;
are abstract representations of features within the raw audio and video data; and
are incapable of being used to reconstruct the raw audio and video data;
storing the raw audio and video data in a secure, short-term ring buffer memory of the local computing device;
based on a first policy, automatically discarding the raw audio and video data from the ring buffer within 400 to 800 milliseconds after the foregoing step of the capture;
retaining the one or more non-reversible embeddings in a non-transitory computer-readable medium after the raw audio and video data has been discarded;
upon receiving an override trigger signal from an override engine;
suspending the first policy to allow controlled access to the raw audio and video data stored in the ring buffer; and
generating a cryptographically signed audit log entry documenting the override trigger signal and the controlled access to the raw audio and video data.
2 . The method of claim 1 , wherein
processing the stream of raw audio and video data using an on-device embedding generator to generate one or more non-reversible, task-specific multimodal embeddings,
further comprising
processing the multimodal embeddings to detect events in a privacy-critical environment, including elder care facilities, mental health centers, and hospitals.
3 . The method of claim 1 , further comprising:
transmitting the one or more non-reversible embeddings to a local processor for inference, while no raw audio and video data is transmitted to the local processor.
4 . The method of claim 1 , further comprising
receiving an authenticated override trigger, including at least one of a gesture, a voice command, or an authenticated API call, to permit temporary storage of the raw audio and video signals in the ring buffer, wherein authentication of the override trigger is verified through a hardware-based mechanism.
5 . The method of claim 1 , wherein
the first policy is enforced by a policy engine executed by a secure enclave or Trusted Platform Module (TPM) of the local computing device.
6 . The method of claim 1 , wherein
automatically discarding the raw audio and video data is enforced by
a watchdog service so that the ring buffer storage time is timed and the watchdog service is executed by the one or more processors of the local computing device.
7 . The method of claim 1 , further comprising
applying a privacy-enhancing technique to the one or more non-reversible embeddings prior to retention, wherein the privacy-enhancing technique comprises at least one of: cryptographic hashing or injection of differential privacy noise.
8 . The method of claim 1 , wherein
the cryptographically signed audit log entry is stored in an immutable data structure.
9 . The method of claim 1 , further comprising
encrypting the raw radio and audio data stored in the ring buffer using a key managed by a secure enclave or Trusted Platform Module (TPM) of the local computing device.
10 . The method of claim 1 , further comprising
detecting events that include multimodal indicators, such as sudden loud audio patterns combined with rapid visual motion.
11 . The method of claim 1 , wherein
the one or more non-reversible, task-specific embeddings comprise one or more of:
an intent embedding, an emotion embedding, an audio event embedding, or a visual event embedding, and wherein processing the stream of the raw audio and video data comprising generating a joint embedding vector using a time-aligned fusion layer with attention mechanisms.
12 . A system for privacy-preserving processing of audio-video data, the system comprising:
at least one processor of a computing device; and
a non-transitory computer-readable storage medium comprising instructions that, when executed by the at least one processor, cause the system to:
capture, via at least one sensor, a stream of raw audio and video data;
process the stream of raw audio and video data using an on-device embedding model to generate one or more non-reversible, task-specific embeddings, wherein the embeddings are abstract feature representations incapable of reconstructing the raw audio and video data;
store the raw audio and video data in a secure, short-term ring buffer of the local computing device;
based on a first policy, automatically discard the raw audio and video data from the ring buffer within a configurable discard delay;
retain the one or more non-reversible embeddings in non-transitory computer-readable medium after the raw audio and video data has been discarded;
upon receiving an override trigger signal from an override component, suspend the first policy to permit controlled access to the raw audio and radio data stored in the ring buffer; and
generate a cryptographically signed audit log entry documenting the override trigger signal and the controlled access to the raw audio and radio data.
13 . The system of claim 12 , wherein the configurable discard delay is between 400 and 800 milliseconds.
14 . The system of claim 12 , wherein the first policy is enforced by a policy engine executed within a secure enclave or Trusted Platform Module (TPM) of the computing device.
15 . The system of claim 12 , wherein automatic discarding of the raw AV data is enforced by a watchdog service monitoring a timed buffer that is linked to a hardware component of the computing device.
16 . The system of claim 12 , wherein the at least one sensor comprises an audio-visual input device including at least one of an RGB camera, a stereo camera, a near-infrared sensor, a thermal imager, or a microphone array.
17 . The system of claim 12 , wherein the on-device embedding generator is a deep-learning model including at least one of a convolutional neural network (CNN), a Vision Transformer (ViT), or a hybrid CNN-RNN architecture.
18 . The system of claim 12 , wherein the one or more non-reversible embeddings comprise one or more of:
an intent embedding, an emotion embedding, an audio event embedding, or a visual event embedding, and
wherein generating the embeddings comprises producing a joint embedding vector using a time-aligned fusion layer with attention mechanisms.
19 . The system of claim 12 , wherein the instructions further cause the system to apply a privacy-enhancing technique to the one or more non-reversible embeddings, wherein
the privacy-enhancing technique comprises at least one of: a cryptographic hash function or differential privacy noise injection.
20 . The system of claim 12 , wherein the instructions further cause the system to process the one or more embeddings to detect events in a privacy-critical environment, including elder care, mental health monitoring, or private settings.
21 . The system of claim 20 , wherein
the privacy-critical environment is an elder care facility, and
wherein the event detection comprises at least one of: fall detection, repetitive behavior detection, or verbal distress detection.
22 . The system of claim 12 , wherein the instructions further cause the system to:
perform federated learning or inference by transmitting the one or more non-reversible embeddings to a remote system, wherein no raw audio and video data is transmitted from the local computing device.
23 . The system of claim 12 , wherein the cryptographically signed audit log entry is stored in an immutable data structure, including at least one of: a Merkle tree or a blockchain ledger.
24 . The system of claim 12 , wherein the at least one processor is implemented on an edge computing platform including an NVIDIA Jetson platform, an FPGA, or a system-on-chip (SoC).