Granted Patent B2

US 10,015,397 · App. 14/885,186 · Granted Jul 3, 2018

Inventors: Phillip J. Emmett (Rochester, NY); Raja Bala (Pittsford, NY); Michael R. Furst (Penfield, NY); Lina Fu (Fairport, NY)

Assignee: Xerox Corporation

H04N5/23245G06K9/00355G06K9/00442G06K9/00496G06K9/036G06K9/228G06K9/32

Abstract

Claims (94)

1. A method of creating an electronic file corresponding to a printed artifact, the method comprising:

by a video capture module of a mobile electronic device, capturing a video of a scene that includes a printed artifact, wherein the video comprises a series of image frames, and at least some of the frames include an image of a portion of the printed artifact and a page-turn event;

by a processor, analyzing a plurality of the image frames in the series in real time as the video is captured to identify a suitable instance, wherein identifying the suitable instance comprises identifying a subset of one or more image frames in the series that include an image of a page or side of the printed artifact and that do not exhibit a page turn event by:

determining a stacked blob that is stacked over a number of successive frames, wherein the stacked blob corresponds to a motion in the video,

measuring a size of the stacked blob, and

determining that a frame exhibits a page turn event if the size of the stacked blob is greater than a threshold;

in response to identification of the suitable instance, without requiring user input, automatically causing a photo capture module to capture a still image of the printed artifact so that the still image has a resolution that is higher than a resolution of the image frames from the video; and

saving the still image to a computer-readable file.

2. The method of claim 1 , further comprising:

upon identification of the suitable instance, stopping the video capture module;

after capturing the still image:

stopping the photo capture module,

resuming the video capture module, and

continuing to analyze, by a decision module in real time as the video is captured by the video capture module, image frames to identify an additional suitable instance for still capture;

upon identification of the additional suitable instance, automatically switching back to the photo capture module to capture an additional still image of the printed artifact; and

saving the captured still images in a single computer-readable file.

3. The method of claim 2 , wherein:

the additional suitable instance corresponds with one or more frames that contain a facet of the printed artifact that differs from a facet of the printed artifact that was contained in the previously-captured image; and

saving each captured still image in the single computer-readable file comprises saving still images of each facet of the printed artifact as a multifaceted document.

4. The method of claim 1 , wherein:

presenting, via a display, the still image to a user for editing; and

receiving, via a user interface, one or more edits to one or more of the still images;

wherein saving the still image in the single computer-readable file comprises saving the still image with the received edits.

5. The method of claim 1 , further comprising, after analyzing a particular subset of the image frames:

determining that the particular subset failed to satisfy one or more image quality criteria; and

executing a feedback module that presents, via a user interface in real time while the device continues to capture the video, feedback for modification of a parameter of the device or the scene so that a subsequently captured subset of frames in the video will correspond to a suitable instance that satisfies the one or more image quality criteria.

6. The method of claim 1 , wherein analyzing the image frames as the video is captured to identify a suitable instance of an image frame also comprises:

receiving motion sensor data as the video is captured;

correlating the motion sensor data to the analyzed image frames;

for each analyzed frame, using the correlated motion sensor data to estimate an amount of motion in the analyzed frame; and

considering the analyzed frame to correspond to a suitable instance only if the estimated amount of motion in the analyzed frame is less than a threshold.

7. The method of claim 1 , wherein analyzing the image frames as the video is captured to identify a suitable instance of an image frame that does not exhibit a page turn event comprises:

estimating an amount of motion in the analyzed frame by performing phase correlation in a frequency domain to obtain a global translational motion estimate; and

considering the analyzed frame to correspond to a suitable instance only if the estimated amount of motion in the analyzed frame is less than a threshold.

8. The method of claim 1 , wherein analyzing the image frames as the video is captured to identify a suitable instance that does not include a page turn event comprises, for each analyzed frame:

detecting the presence of a moving object in the analyzed frame by:

dividing the analyzed frame into a set of N×N sub-blocks of pixels, where N equals an integer N>1,

for each sub-block, determining a standard deviation of luminance values of the pixels within each sub-block, and

for each sub-block, comparing the standard deviation with a standard deviation of a corresponding sub-block in a reference frame for which there is known to be no object presence to yield a difference;

if the difference is greater than a threshold, then concluding that the analyzed frame includes the presence of a moving object.

9. The method of claim 1 , wherein analyzing the image frames as the video is captured to identify a suitable instance comprises, for each analyzed frame:

detecting borders of the printed artifact; and

determining that the analyzed frame satisfies one or more image quality criteria if one or more of the following criteria are met:

all borders of the printed artifact are detected,

a shape formed by the borders is substantially a rectangle or square, or

a determined area enclosed by the detected borders relative to a determined area of the video frame is greater than a threshold.

10. The method of claim 9 , wherein detecting borders of the printed artifact comprises using one or more of the following:

a template matching technique;

a Hough Transform technique; or

a constant color border identification technique.

11. The method of claim 1 , wherein analyzing the image frames as the video is captured to identify a suitable instance comprises, for each analyzed frame:

determining whether an image frame has an image quality score that exceeds a threshold image quality score representing machine readability of the printed artifact; and

the method further comprises determining an image quality score for the frame.

12. The method of claim 11 , wherein, for each analyzed frame, determining the image quality score for the frame comprises:

extracting a set of image patches from the frame;

determining a feature vector for each of the patches from the frame;

pooling the determined feature vectors across the frame; and

retrieving a score for the frame by processing the pooled features via a classifier.

13. The method of claim 1 wherein analyzing the plurality of image frames in the series comprises subsampling the series of image frames according to a spatial or temporal sampling criterion so that a representative set of image frames in the video is analyzed.

14. The method of claim 1 , wherein determining the stacked blob comprises:

applying a morphological erosion operator to each of the number of successive frames;

identifying one or more blobs in each of the eroded frames; and

spatially stacking the blobs that are identified from the number of successive frames to determine the stacked blob.

15. The method of claim 1 , wherein when analyzing the image frames, the system applies one or more rules to determine whether the subset satisfies one or more criteria, and the method further comprises updating the rules based on one or more of time, user behavior, and user feedback.

16. A mobile electronic device, comprising:

a processor;

an image sensor; and

a computer-readable memory containing programming instructions that, when executed, are operable to instruct the processor to:

execute a video capture module that causes the image sensor to capture a video of a scene that includes a printed artifact, wherein the video comprises a series of image frames, and at least some of the frames include an image of a portion of the printed artifact and a page-turn event;

execute a decision module that analyzes a plurality of the image frames in the series in real time as the video is captured to identify a suitable instance, wherein identifying the suitable instance comprises identifying a subset of one or more image frames in the series that include an image of a page or side of the printed artifact and that do not exhibit a page turn event by:

determining a stacked blob that is stacked over a number of successive frames, wherein the stacked blob corresponds to a motion in the video,

measuring a size of the stacked blob, and

determining that a frame exhibits a page turn event if the size of the stacked blob is greater than a threshold;

in response to identification of the suitable instance, without requiring user input, execute a photo capture module that automatically captures a still image of the printed artifact so that the still image has a resolution that is higher than a resolution of the suitable image frame from the video; and

save the still image to a computer-readable file.

17. The system of claim 16 , wherein the instructions further comprise instructions to:

after capturing the still image, stop the photo capture module, and continue to analyze image frames in real time as the video is captured by the video capture module to identify an additional suitable instance for still capture;

upon identification of the additional suitable instance, automatically switch back to the photo capture module to capture an additional still image of the printed artifact; and

save the captured still images in one or more computer-readable files.

18. The system of claim 16 , further comprising a user interface, and wherein the instructions further comprise instructions to execute a feedback module that presents, via the user interface in real time while the device continues to capture the video, feedback to help a user of the device understand what operational criteria are likely to result in a suitable instance.

19. The system of claim 16 , further comprising a motion sensor, and wherein the instructions further comprise instructions to:

correlate the motion sensor data to the analyzed image frames;

for each analyzed frame, use the correlated motion sensor data to estimate an amount of motion in the analyzed frame; and

consider the analyzed frame to be a suitable instance only if the estimated amount of motion in the frame is less than a threshold.

20. The system of claim 16 , wherein the instructions to execute a decision module that analyzes a plurality of the image frames comprise instructions to, for each analyzed frame:

detect borders of the printed artifact; and

determine that the analyzed frame satisfies one or more image quality criteria if one or more of the following criteria are met:

all borders of the printed artifact are detected,

a shape formed by the borders is substantially a rectangle or square, or

a determined area enclosed by the detected borders relative to a determined area of the video frame is greater than a threshold.

21. The system of claim 16 , wherein the instructions further comprise:

instructions to determine whether an image frame has an image quality score that exceeds a threshold image quality score representing machine readability of the printed artifact; and

instructions to, for each analyzed frame, determine an image quality score for the frame.

22. The system of claim 16 , wherein the instructions to analyze the plurality of image frames in the series further comprise instructions to subsample the series of image frames according to a spatial or temporal sampling criterion so that a representative set of image frames in the video are analyzed.

Assignments (8)

Continuity (2)