Systems and methods for body-driven interactions in three-dimension layered images
Methods and systems disclosed herein relate generally to body-driven interactions with three-dimensional (3D) layered graphics. The system includes a video capture module that can receive a video stream. The video stream may depict a subject and a 3D layered image, in which the 3D layered image has an associated viewpoint. The system may also include a video processing module that can identify one or more actions performed by the subject. The video processing module can determine a transform operation to be applied to the viewpoint. The transform operation may include at least one of changing the zoom level of the viewpoint, moving the location of the viewpoint, and changing the direction of the viewpoint. The video processing module may apply the transform operation to the 3D layered image and then render the transformed 3D layered image on the video stream.
1 . A computer-implemented method comprising:
receiving, by a video capture module, a video stream being presented on a display device, the video stream depicting a subject and one or more three-dimensional (3D) layered images in a 3D space, each 3D layered image comprising one or more two dimensional (2D) layers and having an associated viewpoint, each viewpoint having a location, a direction, and a zoom level in the 3D space;
applying, by a video processing module, a machine-learning model to the video stream to identify one or more actions performed by the subject superimposed or embedded within a particular 3D layered image of the one or more 3D layered images;
determining, from the one or more actions, a transform operation to the particular 3D layered image, the particular 3D layered image having a first viewpoint, wherein the transform operation updates the first viewpoint to a second viewpoint, wherein updating the first viewpoint includes at least one of:
changing the zoom level of the first viewpoint to a zoom level of the second viewpoint;
moving the location of the first viewpoint to a location of the second viewpoint; and
changing the direction of the first viewpoint to a direction of the second viewpoint;
applying the transform operation to the particular 3D layered image to generate a transformed 3D layered image, wherein the transformed 3D layered image is configured to be rendered from the perspective of the second viewpoint; and
rendering the transformed 3D layered image on the video stream.
2 . The method of claim 1 , further comprising:
identifying, by the video processing module, a trigger, wherein:
the trigger precedes the identification of the one or more actions performed by the subject; and
the trigger causes the machine-learning model to identify an action of the subject.
3 . The method of claim 2 , wherein the trigger is an input device communicatively coupled with the video processing module.
4 . The method of claim 2 , wherein the trigger is a voice command.
5 . The method of claim 1 , wherein the one or more actions comprise a change in distance between a subject's hands and wherein the transform operation comprises:
changing the zoom level of the first viewpoint, the change in zoom level corresponding to the difference between a first distance between the subject's hands and a second distance between the subject's hands.
6 . The method of claim 1 , wherein the one or more actions comprise a change in position of a subject's head and wherein the transform operation comprises:
changing the direction of the first viewpoint, the change in direction corresponding to the orientation of the subject's head.
7 . The method of claim 1 , wherein the one or more actions comprise:
a change in a first distance on a subject's body; and
a change in a second distance on the subject's body;
and wherein the transform operation comprises:
changing the zoom level of the first viewpoint, the change in zoom level corresponding to the difference between the first distance on the subject's body and the second distance on the subject's body.
8 . The method of claim 7 , wherein the one or more actions further comprise determining a distance change ratio and wherein the changing the zoom level of the first viewpoint is responsive to the distance change ratio exceeding a pre-set threshold.
9 . The method of claim 1 , wherein the video processing module is further configured to receive a set of mappings, the mappings relating a set of voice commands to a set of viewpoint locations, directions, and zoom levels, and wherein:
the one or more actions comprise a voice command from the subject; and
the transform operation comprises at least one of:
changing the direction of the first viewpoint, the change in direction corresponding to the mappings; and
changing the location of the first viewpoint, the change in direction corresponding to the mappings.
10 . A system comprising:
a video capture module configured to:
receive a video stream being presented on a display device, the video stream depicting a subject and one or more three-dimensional (3D) layered images in a 3D space, each 3D layered image comprising one or more 2D layers and having an associated viewpoint, each viewpoint having a location, a direction, and a zoom level in the 3D space; and
a video processing module configured to:
apply a machine-learning model to the video stream to identify one or more actions performed by the subject superimposed or embedded within a particular 3D layered image of the one or more 3D layered images;
determine, from the one or more actions, a transform operation to the particular 3D layered image, the particular 3D layered image having a first viewpoint, wherein the transform operation updates the first viewpoint to a second viewpoint, wherein updating the first viewpoint includes at least one of:
changing the zoom level of the first viewpoint to a zoom level of the second viewpoint;
moving the location of the first viewpoint to a location of the second viewpoint; and
changing the direction of the first viewpoint to a direction of the second viewpoint;
apply the transform operation to the particular 3D layered image to generate a transformed 3D layered image, wherein the transformed 3D layered image is configured to be rendered from the perspective of the second viewpoint; and
render the transformed 3D layered image on the video stream.
11 . The system of claim 10 , wherein the video processing module is further configured to:
identify a trigger, wherein:
the trigger precedes the identification of the one or more actions performed by the subject; and
the trigger causes the machine-learning model to identify an action of the subject.
12 . The system of claim 10 , wherein the one or more actions comprise a change in distance between a subject's hands and wherein the transform operation comprises:
changing the zoom level of the first viewpoint, the change in zoom level corresponding to the difference between a first distance between the subject's hands and a second distance between the subject's hands.
13 . The system of claim 10 , wherein the one or more actions comprise a change in position of a subject's head and wherein the transform operation comprises:
changing the direction of the first viewpoint, the change in direction corresponding to the orientation of the subject's head.
14 . The system of claim 10 , wherein the one or more actions comprise:
a change in a first distance on a subject's body; and
a change in a second distance on the subject's body;
and wherein the transform operation comprises:
changing the zoom level of the first viewpoint, the change in zoom level corresponding to the difference between the first distance on the subject's body and the second distance on the subject's body.
15 . The system of claim 10 , wherein the video processing module is further configured to receive a set of mappings, the mappings relating a set of voice commands to a set of viewpoint locations, directions, and zoom levels, and wherein:
the one or more actions comprise a voice command from the subject; and
the transform operation comprises at least one of:
changing the direction of the first viewpoint, the change in direction corresponding to the mappings; and
changing the location of the first viewpoint, the change in direction corresponding to the mappings.
16 . A computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more processing devices to perform actions including:
receiving a video stream being presented on a display device, the video stream depicting a subject and one or more three-dimensional (3D) layered images in a 3D space, each 3D layered image comprising one or more 2D layers and having an associated viewpoint, each viewpoint having a location, a direction, and a zoom level in the 3D space;
applying a machine-learning model to the video stream to identify one or more actions performed by the subject superimposed or embedded within a particular 3D layered image of the one or more 3D layered images;
determining, from the one or more actions, a transform operation to the particular 3D layered image, the particular 3D layered image having a first viewpoint, wherein the transform operation updates the first viewpoint to a second viewpoint, wherein updating the first viewpoint includes at least one of:
changing the zoom level of the first viewpoint to a zoom level of the second viewpoint;
moving the location of the first viewpoint to a location of the second viewpoint; and
changing the direction of the first viewpoint to a direction of the second viewpoint;
applying the transform operation to the particular 3D layered image to generate a transformed 3D layered image, wherein the transformed 3D layered image is configured to be rendered from the perspective of the second viewpoint; and
rendering the transformed 3D layered image on the video stream.
17 . The computer program product of claim 16 , wherein the one or more actions comprise a change in distance between a subject's hands and wherein the transform operation comprises:
changing the zoom level of the first viewpoint, the change in zoom level corresponding to the difference between a first distance between the subject's hands and a second distance between the subject's hands.
18 . The computer program product of claim 16 , wherein the one or more actions comprise a change in position of a subject's head and wherein the transform operation comprises:
changing the direction of the first viewpoint, the change in direction corresponding to the orientation of the subject's head.
19 . The computer program product of claim 16 , wherein the one or more actions comprise:
a change in a first distance on a subject's body; and
a change in a second distance on the subject's body;
and wherein the transform operation comprises:
changing the zoom level of the first viewpoint, the change in zoom level corresponding to the difference between the first distance on the subject's body and the second distance on the subject's body.
20 . The computer program product of claim 16 , including further instructions to receive a set of mappings, the mappings relating a set of voice commands to a set of viewpoint locations, directions, and zoom levels, and wherein:
the one or more actions comprise a voice command from the subject; and
the transform operation comprises at least one of:
changing the direction of the first viewpoint, the change in direction corresponding to the mappings; and
changing the location of the first viewpoint, the change in direction corresponding to the mappings.