Systems and methods for a multi-resolution visual sensing architecture for augmented reality
A system defining a visual sensing pipeline architecture flexibly allows application developers to dynamically adapt the spatial resolution and update rate of different “rhythmic pixel encoding regions” in a scene. The system ingests pixel streams with raster-scan pixel read-out patterns, but only encodes relevant pixels prior to storing them in memory. The system includes streaming hardware to decode the stored rhythmic pixel encoding region stream into traditional frame-based representations to feed into standard computer vision algorithms.
1. A system, comprising:
a processor in communication with a memory, the memory including instructions, which, when executed, cause the processor to:
receive, at an encoder formulated at the processor, a frame of a plurality of frames, the frame having a plurality of pixels;
generate, by the encoder, an encoded frame having a plurality of regional pixels from the frame and based on a set of encoding region labels, the set of encoding region labels defining one or more encoding regions, a stride resolution, and a skip rate and the plurality of regional pixels including a subset of the plurality of pixels that fall within the one or more encoding regions;
write the encoded frame to the memory preserving raster-scan order; and
write a set of encoding metadata associated with the encoded frame to the memory, the set of encoding metadata including an encoding mask and a per-row offset,
wherein the memory includes further instructions to reconstruct, by a decoder, the frame using the encoded frame, per-row offset and encoding mask.
2. The system of claim 1 , the memory further including instructions executable by the processor to:
receive the set of encoding region labels indicative of an encoding region of the one or more encoding regions, including:
coordinates of a corner of the encoding region;
dimensions of the encoding region;
the stride resolution indicative of a density of pixels of the encoding region to be encoded; and
the skip rate indicative of an interval for consecutive sampling within the encoding region.
3. The system of claim 1 , the memory further including instructions executable by the processor to:
receive data indicative of the frame as a pixel stream at the encoder, the pixel stream including the plurality of pixels of the frame, each pixel of the plurality of pixels being associated with a respective pixel address within the frame;
select, at the encoder, one or more encoding region labels of the set of encoding region labels that correspond to the frame based on a vertical position of a pixel of the plurality of pixels and a corresponding vertical range of an encoding region defined within the set of encoding region labels;
compare a horizontal position of the pixel of the plurality of pixels with a corresponding horizontal range of an encoding region defined within the set of encoding region labels; and
encode, based on the comparison, one or more regional pixels of the frame into the encoded frame that fall within the encoding region.
4. The system of claim 3 , the memory further including instructions executable by the processor to:
omit, based on the comparison, one or more non-regional pixels of the frame that do not fall within the encoding region from the encoded frame;
omit, based on the comparison, one or more strided pixels of the frame from the encoded frame based on the stride resolution defined within the set of encoding region labels;
omit, based on the comparison, one or more skipped pixels of the frame from the encoded frame based on the skip rate defined within the set of encoding region labels; and
generate, based on the comparison, the encoding mask indicating positions of the one or more regional pixels, the one or more skipped pixels, the one or more strided pixels, and the one or more non-regional pixels.
5. The system of claim 3 , the memory further including instructions executable by the processor to:
increment, for each regional pixel within a row of the encoded frame, the per-row offset indicative of a total quantity of regional pixels within the row.
6. The system of claim 1 , the memory further including instructions executable by the processor to:
receive, at the decoder, a request for pixel data including a pixel request address for a pixel of the frame;
translate, at the decoder, the pixel request address to an encoded pixel request address of the encoded frame;
retrieve, based on the encoded pixel request address, encoded pixel response data indicative of the encoded frame;
retrieve, based on the encoded pixel request address, the set of encoding metadata associated with the encoded frame including the per-row offset and the encoding mask; and
generate, at the decoder, output data indicative of a pixel value for each pixel in the frame using the encoding mask.
7. The system of claim 6 , wherein translating the pixel request address to the encoded pixel request address of the encoded frame incorporates a set of encoding metadata associated with one or more previous frames.
8. The system of claim 1 , wherein the decoder includes a first-in, first-out (FIFO) sampling unit in communication with a resampling buffer that interpolates pixel data for reconstruction of the frame based on the encoding mask.
9. The system of claim 8 , wherein the encoding mask indicates that a pixel of the plurality of pixels of the frame is a regional pixel, and wherein the memory further includes instructions executable by the processor to:
de-queue encoded pixel response data from the FIFO sampling unit following retrieval of encoded pixel response data from the memory to obtain a pixel value for the pixel; and
assign the pixel value for the pixel based on the encoded pixel response data.
10. The system of claim 8 , wherein the encoding mask indicates that a pixel of the plurality of pixels of the frame is a strided pixel or a skipped pixel, and wherein the memory further includes instructions executable by the processor to:
re-sample a previous pixel to obtain a pixel value for the pixel; and
assign the pixel value for the pixel based on the previous pixel.
11. The system of claim 8 , wherein the encoding mask indicates that a pixel of the plurality of pixels of the frame is a non-regional pixel, and wherein the memory further includes instructions executable by the processor to:
assign a pixel value for the pixel to be a default pixel value.
12. A system, comprising:
a processor in communication with a memory, the memory including instructions, which, when executed, cause the processor to:
receive, at a decoder in communication with the memory and formulated at the processor, a request for pixel data including a pixel request address for a pixel of a frame, the frame being encoded as an encoded frame and being associated with a set of encoding metadata including a per-row offset and an encoding mask;
translate, at the decoder, the pixel request address to an encoded pixel request address of the encoded frame,
wherein translating the pixel request address to the encoded pixel request address of the encoded frame incorporates a set of encoding metadata associated with one or more previous frames;
retrieve, based on the encoded pixel request address, encoded pixel response data indicative of the encoded frame;
retrieve, based on the encoded pixel request address, the set of encoding metadata associated with the encoded frame including the per-row offset and the encoding mask; and
reconstruct, by the decoder, the frame using the encoded frame, per-row offset and encoding mask.
13. The system of claim 12 , wherein the decoder includes a first-in, first-out (FIFO) sampling unit in communication with a resampling buffer that interpolates pixel data for reconstruction of the frame based on the encoding mask.
14. The system of claim 13 , wherein the encoding mask indicates that a pixel of the frame is a regional pixel, and wherein the memory further includes instructions executable by the processor to:
de-queue encoded pixel response data from the FIFO sampling unit following retrieval of encoded pixel response data from the memory to obtain a pixel value for the pixel; and
assign the pixel value for the pixel based on the encoded pixel response data.
15. The system of claim 13 , wherein the encoding mask indicates that a pixel of the frame is a strided pixel or a skipped pixel, and wherein the memory further includes instructions executable by the processor to:
re-sample a previous pixel to obtain a pixel value for the pixel; and
assign the pixel value for the pixel based on the previous pixel.
16. The system of claim 13 , wherein the encoding mask indicates that a pixel of the frame is a non-regional pixel, and wherein the memory further includes instructions executable by the processor to:
assign a pixel value for the pixel to be a default pixel value.
17. A method, comprising:
receiving, at an encoder formulated at a processor, a frame of a plurality of frames, the frame having a plurality of pixels, further including
receiving the frame as a pixel stream at the encoder, the pixel stream including the plurality of pixels of the frame, each pixel of the plurality of pixels being associated with a respective pixel address within the frame;
generating, by the encoder, an encoded frame having a plurality of regional pixels from the frame and based on a set of encoding region labels, the set of encoding region labels defining one or more encoding regions, a stride resolution, and a skip rate and the plurality of regional pixels including a subset of the plurality of pixels that fall within the one or more encoding regions,
including
selecting, at the encoder, one or more encoding region labels of the set of encoding region labels that correspond to the frame based on a vertical position of a pixel of the plurality of pixels and a corresponding vertical range of an encoding region defined within the set of encoding region labels;
comparing a horizontal position of the pixel of the plurality of pixels with a corresponding horizontal range of an encoding region of the one or more encoding regions defined within the set of encoding region labels; and
encoding, based on the comparison, one or more regional pixels of the frame into the encoded frame that fall within the encoding region;
writing the encoded frame to a memory preserving raster-scan order; and
writing a set of encoding metadata associated with the encoded frame to the memory, the set of encoding metadata including an encoding mask and a per-row offset.
18. The method of claim 17 , further comprising:
omitting, based on the comparison, one or more non-regional pixels of the frame that do not fall within the encoding region from the encoded frame;
omitting, based on the comparison, one or more strided pixels of the frame from the encoded frame based on the stride resolution defined within the set of encoding region labels;
omitting, based on the comparison, one or more skipped pixels of the frame from the encoded frame based on the skip rate defined within the set of encoding region labels; and
generating, based on the comparison, the encoding mask indicating positions of the one or more regional pixels, the one or more skipped pixels, the one or more strided pixels, and the one or more non-regional pixels.
19. The method of claim 17 , further comprising:
reconstructing, by a decoder in communication with the memory and formulated at the processor, the frame using the encoded frame, per-row offset and encoding mask.
20. The method of claim 19 , further comprising:
receiving, at the decoder, a request for pixel data including a pixel request address for a pixel of the frame;
translating, at the decoder, the pixel request address to an encoded pixel request address of the encoded frame using a set of encoding metadata associated with one or more previous frames;
retrieving, based on the encoded pixel request address, encoded pixel response data indicative of the encoded frame;
retrieving, based on the encoded pixel request address, the set of encoding metadata associated with the encoded frame including the per-row offset and the encoding mask; and
generating, at the decoder, output data indicative of a pixel value for each pixel in the frame using the encoding mask.
21. The method of claim 19 , wherein the encoding mask indicates that a pixel of the plurality of pixels of the frame is a regional pixel, the method further comprising:
de-queueing encoded pixel response data from a first-in, first-out (FIFO) sampling unit following retrieval of encoded pixel response data to obtain a pixel value for the pixel; and
assigning the pixel value for the pixel based on the encoded pixel response data.
22. The method of claim 19 , wherein the encoding mask indicates that a pixel of the plurality of pixels of the frame is a strided pixel or a skipped pixel, the method further comprising:
re-sampling a previous pixel to obtain a pixel value for the pixel; and
assigning the pixel value for the pixel based on the previous pixel.
23. The method of claim 19 , wherein the encoding mask indicates that a pixel of the plurality of pixels of the frame is a non-regional pixel, the method further comprising:
assigning a pixel value for the pixel to be a default pixel value.
24. A method, comprising:
receiving, at an encoder formulated at a processor, a frame of a plurality of frames, the frame having a plurality of pixels;
generating, by the encoder, an encoded frame having a plurality of regional pixels from the frame and based on a set of encoding region labels, the set of encoding region labels defining one or more encoding regions, a stride resolution, and a skip rate and the plurality of regional pixels including a subset of the plurality of pixels that fall within the one or more encoding regions, including
incrementing, for each regional pixel within a row of the encoded frame, the per-row offset indicative of a total quantity of regional pixels within the row
writing the encoded frame to a memory preserving raster-scan order; and
writing a set of encoding metadata associated with the encoded frame to the memory, the set of encoding metadata including an encoding mask and a per-row offset.