System and method for 3D tracking for ad-hoc cross-device interaction
Disclosed is a three-dimensional ad-hoc tracking system and method between two or more devices at a location. The disclosed systems and methods can be implemented on commodity mobile devices with no added components and require no cross-device calibration, in order to track surrounding devices. Such tracking can be achieved by fusing three types of signals: 1) the strength of Bluetooth low energy signals reveals the presence and rough distance between devices; 2) a series of inaudible acoustic signals and the difference in their arrival times produces a set of accurate distances between devices, from which 3D offsets between the devices can be derived; and 3) the integration of dead reckoning from the orientation and acceleration sensors of all devices at a location to refine the estimate and to support quick interactions between devices. The disclosed systems and methods can be implemented by cross-device applications on mobile devices.
1. A method comprising:
receiving, at a computing device, Bluetooth Low Energy (BLE) signals from devices at a location;
identifying, via the computing device, said devices as being concurrently present at the location based on the received BLE signals;
determining, via the computing device, a BLE signal strength associated with each identified device based on the received BLE signals;
determining, via the computing device, an audio-distance measurement associated with each identified device, said audio-distance measurement based on acoustic signals and time stamps communicated to and from each identified device;
receiving, at the computing device, an inertia measurement from each device; and
determining, via the computing device, a three-dimensional (3D) spatial mapping of said location based on, for each identified device, the BLE signal strength, the audio-distance measurement and the inertia measurement, said 3D spatial mapping comprising identifiers and positional information of each identified device within said location.
2. The method of claim 1 , wherein said determination of the 3D spatial mapping comprises fusing said BLE signal strength, audio-distance measurement and the inertia measurement of each device.
3. The method of claim 1 , further comprising:
broadcasting, via the computing device, said determined 3D spatial mapping of the location to each identified device at said location.
4. The method of claim 1 , wherein said inertia measurement for each device is based on a constant rotational reference associated with said location, said rotational reference is associated with a magnetometer associated with said location.
5. The method of claim 1 , wherein said inertia measurement for each device comprises at least one of orientation and motion information.
6. The method of claim 1 , wherein said determining said audio-distance measurement based on said acoustic signals comprises:
receiving, by at least one microphone of the computing device, a first acoustic signal from a microphone of each identified device; and
communicating, from at least one speaker of the computing device, a second acoustic signal for reception by a microphone of each identified device.
7. The method of claim 6 , further comprising:
receiving, at the computing device, a time stamp associated with the second acoustic signal from each identified device, said second acoustic signal timestamp associated with a time the second acoustic signal was received by each identified device; and
communicating, via the computing device, a time stamp associated with the first acoustic signal to each identified device, said first acoustic signal time stamp associated with a time the first acoustic signal was received by the computing device.
8. The method of claim 1 , wherein said BLE signal strength determination is based on said determined audio-distance measurement, wherein said BLE signal strength for an identified device is refined in accordance with an audio-distance measurement associated with said identified device.
9. The method of claim 1 , wherein each identified device is within a predetermined distance to the computing device within said location.
10. The method of claim 1 , further comprising:
receiving, at the computing device, a selection of a device identified in said 3D spatial mapping, said selection enabling communication with the selected device.
11. The method of claim 10 , wherein said communication occurs over at least one of a network associated with said location and via a BLE signal packet communicated with the selected device.
12. The method of claim 10 , further comprising:
determining, via the computing device, the 3D spatial mapping of said location based on, for each identified device, the audio-distance measurement.
13. The method of claim 1 , wherein said 3D spatial mapping is updated upon a change in said positional information of at least one device determined to be at said location.
14. A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a processor associated with a computing device, performs a method comprising:
receiving Bluetooth Low Energy (BLE) signals from devices at a location;
identifying said devices as being concurrently present at the location based on the received BLE signals;
determining a BLE signal strength associated with each identified device based on the received BLE signals;
determining an audio-distance measurement associated with each identified device, said audio-distance measurement based on acoustic signals and time stamps communicated to and from each identified device;
receiving an inertia measurement from each device;
determining a three-dimensional (3D) spatial mapping of said location based on, for each identified device, the BLE signal strength, the audio-distance measurement and the inertia measurement, said 3D spatial mapping comprising identifiers and positional information of each identified device within said location.
15. The non-transitory computer-readable storage medium of claim 14 , wherein said determination of the 3D spatial mapping comprises fusing said BLE signal strength, audio-distance measurement and the inertia measurement of each device.
16. The non-transitory computer-readable storage medium of claim 14 , further comprising:
broadcasting said determined 3D spatial mapping of the location to each identified device at said location; and
receiving a selection of a device identified in said 3D spatial mapping, said selection enabling communication with the selected device.
17. The non-transitory computer-readable storage medium of claim 14 , further comprising:
wherein said determining said audio-distance measurement based on said acoustic signals comprises:
receiving a first acoustic signal from at least one microphone of each identified device; and
communicating a second acoustic signal for reception by at least one microphone of each identified device;
receiving a time stamp associated with the second acoustic signal from each identified device, said second acoustic signal timestamp associated with a time the second acoustic signal was received by each identified device; and
communicating a time stamp associated with the first acoustic signal to each identified device, said first acoustic signal time stamp associated with a time the first acoustic signal was received by the computing device.
18. The non-transitory computer-readable storage medium of claim 14 , wherein said BLE signal strength determination is based on said determined audio-distance measurement, wherein said BLE signal strength for an identified device is refined in accordance with an audio-distance measurement associated with said identified device.
19. A computing device comprising:
a processor;
a non-transitory computer-readable storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising:
receiving logic executed by the processor for receiving Bluetooth Low Energy (BLE) signals from devices at a location;
identification logic executed by the processor for identifying said devices as being concurrently present at the location based on the received BLE signals;
determination logic executed by the processor for determining a BLE signal strength associated with each identified device based on the received BLE signals;
determination logic executed by the processor for determining an audio-distance measurement associated with each identified device, said audio-distance measurement based on acoustic signals and time stamps communicated to and from each identified device;
receiving logic executed by the processor for receiving an inertia measurement from each device;
determination logic executed by the processor for determining a three-dimensional (3D) spatial mapping of said location based on, for each identified device, the BLE signal strength, the audio-distance measurement and the inertia measurement, said 3D spatial mapping comprising identifiers and positional information of each identified device within said location; and
communication logic executed by the processor for broadcasting said determined 3D spatial mapping of the location to each identified device at said location.