Marker based tracking
A tracking system converts images to a set of points in 3D space. The images are of a wearable item that includes markers, and the set of points include representations of the markers. A view is selected from a plurality of views using the set of points, and the selected view includes one or more representations of the representations. A depth map is generated based on the selected view and the set of points, and the depth map includes the one or more representations. A neural network maps labels to the one or more representations in the depth map using a model of a portion of a body that wears the wearable item. A joint parameter is determined using the mapped labels. The model is updated with the joint parameter, and content provided to a user of the wearable item is based in part on the updated model.
1. A method comprising:
converting images to a set of points in 3D space, wherein the images are of a wearable item that includes markers, and the set of points in 3D space include representations of the markers;
selecting a view from a plurality of views using the set of points, wherein the selected view includes one or more representations of the representations;
generating a depth map based on the selected view and the set of points, wherein the depth map includes the one or more representations;
mapping, via a neural network, labels to the one or more representations in the depth map using a model of a portion of a body that wears the wearable item;
determining a joint parameter using the mapped labels; and
updating the model with the joint parameter, wherein content provided to a user of the wearable item is based in part on the updated model.
2. The method of claim 1 , wherein the mapping, via the neural network, labels to the one or more representations in the depth map using the model of the portion of the body that wears the wearable item, further comprises:
mapping positions of joints described in the model to image space positions using the neural network;
matching the image space positions of the joints to image space positions of the markers; and
labeling the matched positions of the joints.
3. The method of claim 2 , wherein the matching the image space positions of the joints to image space positions of the markers is done using minimum—cost bipartite matching.
4. The method of claim 1 , wherein the images of the wearable item includes a plurality of viewpoints of the wearable item.
5. The method of claim 1 , wherein selecting the view from the plurality of views using the set of points comprises:
determining an orientation of a synthetic camera relative to the set of points that includes at least a threshold number of representations; and
generating the view based on the determined orientation and a field of view of the synthetic camera.
6. The method of claim 1 , wherein selecting the view from the plurality of views using the set of points comprises:
determining an orientation, of a plurality of orientations, of a synthetic camera relative to the set of points that includes a largest number of representations; and
generating the view based on the determined orientation and a field of view of the synthetic camera.
7. The method of claim 1 , wherein the neural network is a convolutional neural network.
8. The method of claim 1 , wherein the wearable item is a glove.
9. The method of claim 1 , further comprising:
capturing, via a plurality of cameras, the images of the wearable item, wherein the captured images include at least one of the markers.
10. The method of claim 9 , wherein the plurality of cameras each have a different field of view than any of the plurality of views.
11. A non-transitory computer-readable medium having embedded thereon a program, the program being executable by a processor for performing a method comprising:
converting images to a set of points in 3D space, wherein the images are of a wearable item that includes markers, and the set of points in 3D space include representations of the markers;
selecting a view from a plurality of views using the set of points, wherein the selected view includes one or more representations of the representations;
generating a depth map based on the selected view and the set of points, wherein the depth map includes the one or more representations;
mapping, via a neural network, labels to the one or more representations in the depth map using a model of a portion of a body that wears the wearable item;
determining a joint parameter using the mapped labels; and
updating the model with the joint parameter, wherein content provided to a user of the wearable item is based in part on the updated model.
12. The computer-readable medium of claim 11 , wherein the mapping, via the neural network, labels to the one or more representations in the depth map using the model of the portion of the body that wears the wearable item, further comprises:
mapping positions of joints described in the model to image space positions using the neural network;
matching the image space positions of the joints to image space positions of the markers; and
labeling the matched positions of the joints.
13. The computer-readable medium of claim 12 , wherein the matching the image space positions of the joints to image space positions of the markers is done using minimum—cost bipartite matching.
14. The computer-readable medium of claim 11 , wherein the images of the wearable item includes a plurality of viewpoints of the wearable item.
15. The computer-readable medium of claim 11 , wherein selecting the view from the plurality of views using the set of points comprises:
determining an orientation of a synthetic camera relative to the set of points that includes at least a threshold number of representations; and
generating the view based on the determined orientation and a field of view of the synthetic camera.
16. The computer-readable medium of claim 11 , wherein selecting the view from the plurality of views using the set of points comprises:
determining an orientation, of a plurality of orientations, of a synthetic camera relative to the set of points that includes a largest number of representations; and
generating the view based on the determined orientation and a field of view of the synthetic camera.
17. The computer-readable medium of claim 11 , wherein the neural network is a convolutional neural network.
18. The computer-readable medium of claim 11 , wherein the wearable item is a glove.
19. The computer-readable medium of claim 11 , further comprising:
capturing, via a plurality of cameras, the images of the wearable item, wherein the captured images include at least one of the markers.
20. The computer-readable medium of claim 19 , wherein the plurality of cameras each have a different field of view than any of the plurality of views.