Object tracking using sparse sensor captures
In one embodiment, a method includes instructing, at a first time, a camera having a plurality of pixel sensors to capture a first image of an environment comprising an object to determine a first object pose; determining, based on the first object pose, a predicted object pose of the object at a second time; generating pixel-activation instructions based on a buffer region around a projection of a 3D model of the object having the predicted object pose onto a virtual image plane associated with a predicted camera pose, where the size of the buffer region may be dependent on predicted dynamics for the object; instructing, at the second time, the camera to use a subset of the plurality of pixel sensors to capture a second image of the environment according to the pixel-activation instructions, and; determining, based on the second image, a second object pose of the object.
1. A method comprising, by one or more computing systems:
instructing, at a first time, a camera having a plurality of pixel sensors to capture a first image of an environment comprising an object to determine a first object pose of the object;
determining, based on the first object pose, a predicted object pose of the object at a second time;
generating pixel-activation instructions based on a buffer region around a projection of a 3D model of the object having the predicted object pose onto a virtual image plane associated with a predicted camera pose, wherein a size of the buffer region is dependent on predicted dynamics for the object;
instructing, at the second time, the camera to use a subset of the plurality of pixel sensors to capture a second image of the environment according to the pixel-activation instructions; and
determining, based on the second image, a second object pose of the object.
2. The method of claim 1 , wherein the subset of the plurality of pixel sensors are activated and a second subset of the plurality of pixel sensors are inactivated at the second time to capture the second image.
3. The method of claim 1 , further comprising:
identifying an object type of the object based on the first image;
accessing a prediction model associated with the object type; and
using the prediction model to determine the predicted object pose of the object at the second time.
4. The method of claim 1 , further comprising:
generating an artificial-reality effect based on the second object pose; and
displaying the artificial-reality effect using a head-mounted device worn by a user;
wherein the camera is attached to the head-mounted device worn by the user.
5. The method of claim 1 , wherein the size of the buffer region is proportional to the time between the first time and the second time.
6. The method of claim 1 , wherein the size of the buffer region is proportional to a latency of the one or more computing systems.
7. The method of claim 1 , wherein the size of the buffer region is proportional to a framerate of the camera.
8. The method of claim 1 , further comprising:
activating all the pixels within the buffer region.
9. The method of claim 1 , further comprising:
activating a sparse sample of pixels within the buffer region.
10. The method of claim 9 , further comprising:
increasing a sampling of pixels within the projection of the 3D model within the buffer region.
11. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
instruct, at a first time, a camera having a plurality of pixel sensors to capture a first image of an environment comprising an object to determine a first object pose of the object;
determine, based on the first object pose, a predicted object pose of the object at a second time;
generate pixel-activation instructions based on a buffer region around a projection of a 3D model of the object having the predicted object pose onto a virtual image plane associated with a predicted camera pose, wherein a size of the buffer region is dependent on predicted dynamics for the object;
instruct, at the second time, the camera to use a subset of the plurality of pixel sensors to capture a second image of the environment according to the pixel-activation instructions; and
determine, based on the second image, a second object pose of the object.
12. The media of claim 11 , wherein the subset of the plurality of pixel sensors are activated and a second subset of the plurality of pixel sensors are inactivated at the second time to capture the second image.
13. The media of claim 11 , wherein the software is further operable when executed to:
identify an object type of the object based on the first image;
access a prediction model associated with the object type; and
use the prediction model to determine the predicted object pose of the object at the second time.
14. The media of claim 11 , wherein the software is further operable when executed to:
generate an artificial-reality effect based on the second object pose; and
display the artificial-reality effect using a head-mounted device worn by a user;
wherein the camera is attached to the head-mounted device worn by the user.
15. The media of claim 11 , wherein the size of the buffer region is proportional to the time between the first time and the second time.
16. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
instruct, at a first time, a camera having a plurality of pixel sensors to capture a first image of an environment comprising an object to determine a first object pose of the object;
determine, based on the first object pose, a predicted object pose of the object at a second time;
generate pixel-activation instructions based on a buffer region around a projection of a 3D model of the object having the predicted object pose onto a virtual image plane associated with a predicted camera pose, wherein a size of the buffer region is dependent on predicted dynamics for the object;
instruct, at the second time, the camera to use a subset of the plurality of pixel sensors to capture a second image of the environment according to the pixel-activation instructions; and
determine, based on the second image, a second object pose of the object.
17. The system of claim 16 , wherein the subset of the plurality of pixel sensors are activated and a second subset of the plurality of pixel sensors are inactivated at the second time to capture the second image.
18. The system of claim 16 , wherein the processors are further operable when executing the instructions to:
identify an object type of the object based on the first image;
access a prediction model associated with the object type; and
use the prediction model to determine the predicted object pose of the object at the second time.
19. The system of claim 16 , wherein the processors are further operable when executing the instructions to:
generate an artificial-reality effect based on the second object pose; and
display the artificial-reality effect using a head-mounted device worn by a user;
wherein the camera is attached to the head-mounted device worn by the user.
20. The system of claim 16 , wherein the size of the buffer region is proportional to the time between the first time and the second time.