Capturing and synchronizing data from multiple sensors
Processes, methods, systems, and devices are disclosed for synchronizing multiple wireless data streams captured in action by various sensors, with lost data recovery. For example, a source device may have multiple sensors acquiring data and sending the data streams (e.g., via Bluetooth connections) to a target device. Timing information may be appended for each of the data streams. Data packets of the multiple data streams may be formed with the timing information. The data packets may be transmitted to a target device that is configured to synchronize the multiple data streams using the timing information. The target device, applying the example processes or techniques of this disclosure, may accurately synchronize the multiple data streams. In some cases, the target device may capture additional data streams and the processor synchronizes all data streams of both the source and the target devices.
1. A method performed by a source device comprising:
acquiring, via a plurality of sensors, a plurality of data streams;
appending timing information for each of the data streams;
forming data packets of the plurality of data streams with the timing information;
transmitting the data packets to a target device configured to synchronize the plurality of data streams using the timing information; and
prioritizing one or more of the plurality of data streams based on an order of reliability received from the target device.
2. The method of claim 1 , wherein the timing information comprises a timestamp for each data packet of the plurality of data streams, wherein the timestamp identifies a shared time instance of a subset of the plurality of data streams.
3. The method of claim 2 , wherein the subset of the plurality of data streams and each data packet of the plurality of data streams have a same time duration.
4. The method of claim 1 , further comprising:
providing a time slot by an initiator device, wherein providing the timing information for each of the plurality of data streams comprises assigning the time slot to each of the plurality of data streams.
5. The method of claim 4 , wherein transmitting the data packets comprises transmitting the data packets using an isochronous channel.
6. The method of claim 1 , wherein transmitting the data packets comprises transmitting the data packets in separate channels, each of the channels associated with at least one sensor configured to capture one of the plurality of data streams.
7. The method of claim 6 , further comprising:
multiplexing the data packets prior to transmitting the data packets in the separate channels.
8. The method of claim 1 , wherein the plurality of sensors includes at least one of a microphone, camera, accelerometer, gyroscope, or inertial measurement unit.
9. A method for providing audio synchronization to a user at a target device, the method comprising:
receiving, from a source device, data packets associated with a plurality of data streams acquired via a plurality of sensors;
decoding the plurality of data streams from the received data packets;
extracting timing information associated with the data packets;
synchronizing the plurality of data streams using the timing information;
determining an order of reliability; and
prioritizing one or more of the plurality of data streams based on the determined order of reliability.
10. The method of claim 9 , further comprising:
providing a user interface on the target device, the user interface capable of accepting user input for processing the received data packets;
providing via the user interface a set of data synchronization and recovery options for the plurality of data streams; and
providing, via the user interface, a capability for the user to launch a data acquisition event initiating a stream of the plurality of data streams.
11. The method of claim 9 , wherein extracting the timing information associated with the data packets comprises processing an appended timestamp for each of the received data packets, the timestamp appended to each data packet of the plurality of data streams and identifying a shared time instance of a subset of the plurality of data streams, wherein the subset of the plurality of data streams and each data packet of the plurality of data streams have a same time duration.
12. The method of claim 9 , wherein extracting the timing information associated with the data packets further comprising identifying a synchronization time slot assigned by an initiator device for each of the plurality of data streams, wherein receiving the packets comprises receiving the packets via an isochronous channel.
13. The method of claim 9 , wherein receiving the data packets comprises receiving the data packets in separate channels, each of the channels associated with at least one sensor configured to capture one of the plurality of data streams; and
the method further comprising reassembling the plurality of data streams using the data packets of the separate channels of sensors.
14. The method of claim 13 , wherein reassembling the plurality of data streams using the data packets of the separate channels of sensors comprises prioritizing at least one of the separate channels.
15. The method of claim 9 , further comprising:
identifying lost data packets based on the extracted timing information;
identifying a location and an amount of lost data packets for each of the plurality of data streams; and
generating a recovery section of data packets to replace the lost data packets, wherein the recovery section of data packets comprises at least one of silent packets, noise packets, or synthesized packets.
16. The method of claim 15 , wherein generating the recovery section of data packets of synthesized packets comprises processing one channel of the plurality of data streams having received data packets at the location of the lost data packets of another channel of the plurality of data streams.
17. The method of claim 16 , wherein processing the one channel of the plurality of data streams having received data packets at the location of the lost data packets of another channel of the plurality of data streams comprises at least one of scaling or extrapolation of the received data packets at the location of the lost data packets.
18. The method of claim 9 , wherein the plurality of sensors includes at least one of a microphone, camera, accelerometer, gyroscope, or inertial measurement unit.
19. A device comprising:
a plurality of sensors configured to acquire a plurality of data streams;
a processor configured to:
append timing information for each of the data streams;
form data packets of the plurality of data streams with the timing information; and
prioritize one or more of the plurality of data streams based on an order of reliability received from a target device; and
a transceiver configured to transmit the data packets to the target device configured to synchronize the plurality of data streams using the timing information.
20. The device of claim 19 , wherein the timing information comprises a timestamp for each data packet of the plurality of data streams, wherein the timestamp identifies a shared time instance of a subset of the plurality of data streams.
21. The device of claim 20 , wherein the subset of the plurality of data streams and each data packet of the plurality of data streams have a same time duration.
22. The device of claim 19 , wherein the plurality of sensors comprises eight microphones positioned to capture a surround-sound.