Automatic segmentation for screen-based tutorials using AR image anchors
Example implementations described herein involve systems and methods for a mobile application device to playback and record augmented reality (AR) overlays indicating gestures to be made to a recorded device screen. A device screen is recorded by a camera of the mobile device, wherein a mask is overlaid on a user hand interacting with the device screen. Interactions made to the device screen are detected based on the mask, and AR overlays are generated corresponding to the reactions.
1. A non-transitory computer readable medium, storing instructions for executing a process for a device comprising a camera and a processor, the instructions comprising:
detecting a device screen from images received from the camera;
upon initiation of a recording of the detected device screen, generating perspective corrected frames of the detected device screen from the images received from the camera; and
generating a mask on gestures made to the detected device screen detected from the images received from the camera, wherein the mask is based on one or more frames from the recording;
processing the recording for reference images of the device screen, and interactions made to the device screen based on the mask;
generating augmented reality (AR) overlays for the reference images based on the interactions made to the device screen based on the mask;
determining a type for each of the AR overlays based on differences between binarized images of the reference images corresponding to each of the interactions; and
determining the type for the each of the AR overlays as a tap gesture overlay for the differences between binarized images being below a threshold.
2. The non-transitory computer readable medium of claim 1 , the instructions further comprising transmitting, to a database, another recording for playback comprising the reference images and the AR overlays.
3. The non-transitory computer readable medium of claim 1 , wherein the processing the recording for reference images of the detected device screen, and interactions made to the detected device screen based on the mask comprises using frames from the recording in which the mask is not overlaid on the detected device screen as the reference images.
4. The non-transitory computer readable medium of claim 1 , wherein the processing the recording for reference images of the detected device screen, and interactions made to the detected device screen based on the mask comprises identifying the interactions from identifying fingertip interactions on the detected device screen.
5. The non-transitory computer readable medium of claim 1 , wherein the generating augmented reality (AR) overlays for the reference images based on the interactions made to the detected device screen based on the mask comprises, for each of the interactions:
determining a location for each of the AR overlays corresponding to the each of the interactions on the reference images based on a location of the each of the interactions; and
generating the each of the AR overlays for the each of the interactions on the reference images at the location on the reference images and according to the type.
6. The non-transitory computer readable medium of claim 1 , wherein the determining the type for the each of the AR overlays based on differences between binarized images of the reference images corresponding to the interactions comprises:
for the differences between binarized images indicating transitions between above the threshold, determining the type for the each of the AR overlays as a swipe gesture overlay.
7. A non-transitory computer readable medium, storing instructions for executing a process for a device comprising a camera and a processor, the instructions comprising:
detecting a device screen from images received from the camera;
retrieving a recording corresponding to the detected device screen;
upon initiation of a playback of the recording corresponding to the detected device screen:
playing the recording corresponding to the detected device screen until an augmented reality (AR) overlay corresponding to an interaction is reached;
stopping the recording until a change is detected on the detected device screen from the images received from the camera; and
continuing playback of the recording once the change is detected on the detected device screen.
8. The non-transitory computer readable medium of claim 7 , wherein the AR overlay indicates one of a tap gesture or a scroll gesture to be made on a location of the detected device screen.
9. A non-transitory computer readable medium, storing instructions for a management apparatus configured to facilitate an application for a mobile device, the instructions comprising:
receiving a recording of a device screen comprising perspective corrected frames of the device screen and a mask on gestures made to the device screen;
processing the recording for reference images of the device screen, and interactions made to the device screen based on the mask, wherein the mask is based on one or more frames from the recording;
generating augmented reality (AR) overlays for the reference images based on the interactions made to the device screen based on the mask;
determining a type for each of the AR overlays based on differences between binarized images of the reference images corresponding to each of the interactions; and
determining the type for the each of the AR overlays as a tap gesture overlay for the differences between binarized images being below a threshold.
10. The non-transitory computer readable medium of claim 9 , the instructions further comprising storing, in a database, another recording for playback comprising the reference images and the AR overlays.
11. The non-transitory computer readable medium of claim 9 , wherein the processing the recording for reference images of the device screen, and interactions made to the device screen based on the mask comprises using frames from the recording in which the mask is not overlaid on the detected device screen as the reference images.
12. The non-transitory computer readable medium of claim 9 , wherein the processing the recording for reference images of the device screen, and interactions made to the device screen based on the mask comprises identifying the interactions from identifying fingertip interactions on the detected device screen.
13. The non-transitory computer readable medium of claim 9 , wherein the generating augmented reality (AR) overlays for the reference images based on the interactions made to the device screen based on the mask comprises, for each of the interactions:
determining a location for each of the AR overlays corresponding to the each of the interactions on the reference images based on a location of the each of the interactions; and
generating the each of the AR overlays for the each of the interactions on the reference images at the location on the reference images and according to the type.
14. The non-transitory computer readable medium of claim 9 , wherein the determining the type for the each of the AR overlays based on differences between binarized images of the reference images corresponding to the interactions comprises:
for the differences between binarized images indicating transitions between above the threshold, determining the type for the each of the AR overlays as a swipe gesture overlay.