Methods and systems for offloading pose processing to a mobile device for motion tracking on a hardware device without a camera
Systems and methods for motion tracking on a hardware device without a camera and/or without a sufficiently powerful hardware processor, by offloading pose processing fully or partially to a mobile device, are disclosed. Methods executable on the hardware device include receiving pose data associated with a user from the mobile device, and performing a touchless control of an application on the hardware device, based on the received pose data. Methods executable on the mobile device include capturing at least one image associated with a user, performing a pose recognition algorithm on the mobile device, based on the captured image, to generate pose data for performing a touchless control of an application on the hardware device, and transmitting the pose data to the hardware device, using the network connection. The methods and systems include embodiments where the pose recognition algorithm uses machine learning.
1. A non-transitory physical storage medium storing program code, the program code executable by a hardware processor, the hardware processor when executing the program code causing the hardware processor to execute a computer-implemented process for motion tracking on a hardware device without a camera and/or without a sufficiently powerful hardware processor, the process executable on a mobile device with a camera by offloading pose processing to the mobile device, the program code comprising code to:
pair the mobile device with the hardware device over a network connection, the network connection for transmitting pose data from the mobile device to the hardware device;
capture at least one image associated with a user;
perform a pose recognition algorithm on the mobile device, based on the captured image, to generate the pose data for performing a touchless control of an application on the hardware device, wherein the pose recognition algorithm further comprises program code to apply a convolutional neural network to the image to determine a feature map; program code to determine a keypoints map and an affinity-field map based on the feature map; and program code to determine the pose data of the user based on the keypoints map and the affinity-field map; and
transmit the pose data to the hardware device over the network connection.
2. The non-transitory physical storage medium of claim 1 , wherein the program code to perform the pose recognition algorithm further comprises program code to:
identify, using an image-recognition technique, an activation gesture of the user in the image;
determine a validity of a bounding box associated with a pose of the user; and
track, using a pose-estimation technique, the pose of the user and a portion of a body of the user within the bounding box, to generate a tracked pose.
3. The non-transitory physical storage medium of claim 1 , wherein the pose data is selected from the group consisting of a feature map, a keypoints map, an affinity-field map, and a tracked pose.
4. The non-transitory physical storage medium of claim 1 , further comprising program code to:
capture at least one video on the mobile device; and
perform a pose recognition algorithm on the mobile device, based on at least one frame of the captured video, to generate the pose data.
5. The non-transitory physical storage medium of claim 1 , wherein the network connection uses a technology selected from the group consisting of a local area network (LAN), a wide area network (WAN), a Bluetooth link, a WiFi network, a sensor network, a cellular network, an ad-hoc wireless network, an enterprise network, and a body area network (BAN).
6. The non-transitory physical storage medium of claim 1 , wherein the pose data is used by the application on the hardware device to perform an operation in the application.
7. A non-transitory physical storage medium storing program code, the program code executable by a hardware processor, the hardware processor when executing the program code causing the hardware processor to execute a computer-implemented process for motion tracking, the process executable on a hardware device without a camera and/or without a sufficiently powerful hardware processor by offloading pose processing to a mobile device with a camera, the program code comprising code to:
pair the hardware device with the mobile device over a network connection, the network connection for transmitting pose data from the mobile device to the hardware device;
receive the pose data associated with a user from the mobile device over the network connection, wherein the pose data was derived using a pose recognition algorithm comprising applying a convolutional neural network to an image of the user to determine a feature map; determining a keypoints map and an affinity-field map based on the feature map; and determining the pose data of the user based on the keypoints map and the affinity-field map; and
perform a touchless control of an application on the hardware device, based on the received pose data.
8. The non-transitory physical storage medium of claim 7 , wherein the program code to perform the touchless control of the application on the hardware device, based on the received pose data, further comprises program code to:
determine, based on the received pose data, a mapping between the portion of the user's body and a user interface element position associated with an application executed on the hardware device; and
move the user interface element position based on the mapping to enable an interaction between the user and the application.
9. The non-transitory physical storage medium of claim 7 , wherein the pose data is selected from the group consisting of a feature map, a keypoints map, an affinity-field map, and a tracked pose.
10. The non-transitory physical storage medium of claim 7 , wherein the network connection uses a technology selected from the group consisting of a local area network (LAN), a wide area network (WAN), a Bluetooth link, a WiFi network, a sensor network, a cellular network, an ad-hoc wireless network, an enterprise network, and a body area network (BAN).
11. The non-transitory physical storage medium of claim 7 , wherein the pose data is used by the application on the hardware device to perform an operation in the application.
12. A motion tracking system, comprising:
a hardware device without a camera and/or without a sufficiently powerful hardware processor;
a mobile device with a camera; and
a network connection for communicating between the hardware device and the mobile device;
at least one non-transitory physical storage medium storing program code, the program code executable by at least one hardware processor, the at least one hardware processor when executing the program code causing the at least one hardware processor to execute a computer-implemented process for motion tracking on the hardware device, the process executable on the mobile device by offloading pose processing to the mobile device, the program code comprising code to:
pair the mobile device with the hardware device over the network connection;
capture at least one image associated with a user at the mobile device;
perform a pose recognition algorithm on the mobile device, based on the captured image, to generate pose data, wherein the pose recognition algorithm further comprises program code to apply a convolutional neural network to the image to determine a feature map; program code to determine a keypoints map and an affinity-field map based on the feature map; and program code to determine the pose data of the user based on the keypoints map and the affinity-field map;
transmit the pose data to the hardware device over the network connection; and
perform a touchless control of an application on the hardware device, based on the pose data received at the hardware device.
13. The motion tracking system of claim 12 , wherein the pose data is selected from the group consisting of a feature map, a keypoints map, an affinity-field map, and a tracked pose.
14. The motion tracking system of claim 12 , further comprising program code to:
capture at least one video on the mobile device; and
perform a pose recognition algorithm on the mobile device, based on at least one frame of the captured video, to generate the pose data.