Lightspill display for pass-through video
Various implementations disclosed herein include devices, systems, and methods that uses blend circuitry to provide a view of live pass-through video with a virtual object including a light-spill effect. For example, a process includes capturing pass-through video of a physical environment. The process further includes capturing virtual content. The process further includes calculating a color from the virtual content. The process further includes controlling the blend circuitry to generate an augmented reality video by blending the pass-through video with the virtual content and by modifying at least one portion of the pass-through video with the calculated color from the virtual content.
1 . A device comprising: at least one camera capturing pass-through video of a physical environment;
a display;
blend circuitry that generates augmented reality video for the display from the pass-through video;
at least one processor; and
a non-transitory computer-readable storage medium comprising instructions that, when executed on the at least one processor, cause the device to perform operations comprising:
producing virtual content corresponding to a virtual light producing object, the virtual content comprising frame-specific virtual content for each of a plurality of frames of the pass-through video;
calculating a color from the virtual content; and
controlling the blend circuitry to generate the augmented reality video by blending the pass-through video with the virtual content and by modifying at least one portion of the pass-through video with the calculated color from the virtual content.
2 . The device of claim 1 , wherein the blend circuitry comprises a dedicated pathway to the at least one camera.
3 . The device of claim 1 , wherein the virtual light producing object is a virtual movie screen or a virtual television.
4 . The device of claim 1 , wherein the virtual light producing object is a virtual 3D object corresponding to a diffuse-light-producing 3D object.
5 . The device of claim 1 , wherein the modifying comprises combining,
via the blend circuitry, each of the frames of the pass-through video with the frame-specific virtual content and altering a portion of each of the frames of the pass-through video to provide a light-spill effect, wherein the light-spill effect is based on at least one color determined from the frame-specific virtual content.
6 . The device of claim 5 , wherein altering the portion of each of the frames of the pass-through video to provide the light-spill effect comprises tinting, dimming, or changing a brightness of a respective portion of each of the frames.
7 . The device of claim 5 , wherein altering the portion of each of the frames of the pass-through video to provide the light-spill effect comprises performing a hardware-implemented logic operation via the blend circuitry.
8 . The device of claim 5 , wherein altering the portion of each of the frames of the pass-through video to provide the light-spill effect comprises utilizing the blend circuitry to apply a tinting, dimming, or brightness adjustment of the portion.
9 . The device of claim 5 , wherein the at least one color determined from the frame-specific virtual content is determined by:
down-sampling the virtual content to produce a blurred image representing the virtual content; and
selecting the at least one color by identifying a dominant color of the down-sampled virtual content.
10 . The device of claim 5 , wherein the at least one color determined from the frame-specific virtual content is determined by:
down-sampling the virtual content to produce a blurred image representing the virtual content; and
selecting the at least one color by identifying at least a first color on a first side of the down-sampled virtual content and at least a second color on a second side of the down-sampled virtual content.
11 . The device of claim 5 , wherein said combining each of the frames of the pass-through video with the frame-specific virtual content comprises utilizing the blend circuitry to enable a hardware-based alpha blending process.
12 . The device of claim 5 , wherein said combining each of the frames of the pass-through video with the frame-specific virtual content comprises adjusting alpha-blend values corresponding to the pass-through video in areas corresponding to the virtual light producing object.
13 . The device of claim 1 , wherein the augmented reality video has a frame rate greater than 60 fps.
14 . The device of claim 1 , wherein the operations further comprise displaying the augmented reality video.
15 . A device comprising: at least one camera capturing pass-through video of a physical environment;
a display;
blend circuitry that generates augmented reality video for the display from the pass-through video;
at least one processor; and
a non-transitory computer-readable storage medium comprising instructions that, when executed on the at least one processor, cause the device to perform operations comprising:
producing virtual content;
calculating a color from the virtual content; and
controlling the blend circuitry to generate the augmented reality video by blending the pass-through video with the virtual content and by modifying at least one portion of the pass-through video with the calculated color from the virtual content, wherein the augmented reality video is provided in approximately real time with the capturing of the pass-through video.
16 . A method comprising:
at a device having, at least one processor, at least one camera, a display, and blend circuitry:
capturing pass-through video of a physical environment via the at least one camera;
producing virtual content corresponding to a virtual light producing object, the virtual content comprising frame-specific virtual content for each of a plurality of frames of the pass-through video;
calculating a color from the virtual content; and
controlling the blend circuitry to generate an augmented reality video by blending the pass-through video with the virtual content and by modifying at least one portion of the pass-through video with the calculated color from the virtual content.
17 . The method of claim 16 , wherein the blend circuitry comprises a dedicated pathway to the at least one camera.
18 . A non-transitory computer-readable storage medium storing program instructions executable via one or more processors, of a device having at least one camera, a display, and blend circuitry, to perform operations comprising:
capturing pass-through video of a physical environment via the at least one camera;
producing virtual content corresponding to a virtual light producing object, the virtual content comprising frame-specific virtual content for each of a plurality of frames of the pass-through video;
calculating a color from the virtual content; and
controlling the blend circuitry to generate an augmented reality video by blending the pass-through video with the virtual content and by modifying at least one portion of the pass-through video with the calculated color from the virtual content.