Live document detection in a captured video stream
View Patent ↗The present disclosure is directed toward systems and methods to quickly and accurately identify boundaries of a displayed document in a live camera image feed, and provide a document boundary indicator within the live camera image feed. For example, systems and methods described herein utilize different display document detection processes in parallel to generate and provide a document boundary indicator that accurately corresponds with a displayed document within a live camera image feed. Thus, a user of the mobile computing device can easily see whether the document identification system has correctly identified the displayed document within the camera viewfinder feed.
1. A method comprising:
receiving an image feed comprising a displayed document;
analyzing, by at least one processor, a first image frame from the image feed using a first process to determine a boundary of the displayed document;
providing, for presentation on a display of a client computing device, the first image frame with a document boundary indicator corresponding to the boundary of the displayed document;
analyzing, based on the boundary of the displayed document in the first image frame and in parallel with the first process, a second image frame from the image feed using a second process to determine an updated boundary of the displayed document; and
providing, for presentation on the display of the client computing device, the second image frame with an updated document boundary indicator corresponding to the updated boundary of the displayed document.
2. The method as recited in claim 1 , wherein receiving the image feed comprises receiving the image feed from a camera associated with the client computing device.
3. The method as recited in claim 1 , wherein analyzing the first image frame using the first process comprises analyzing the first image frame using a robust detection process.
4. The method as recited in claim 3 , wherein analyzing the first image frame using the robust detection process comprises:
generating an edge map comprising edges identified within the first image frame;
identifying, based the edge map, a plurality of edges and a plurality of corners within the first image frame; and
determining one or more quadrilaterals formed from the plurality of edges and the plurality of corners.
5. The method as recited in claim 4 , further comprising:
analyzing the one or more quadrilaterals to assign a score to each of the one or more quadrilaterals; and
wherein determining the boundary of the displayed document is based on identifying a quadrilateral from the one or more quadrilaterals that is associated with the highest score.
6. The method as recited in claim 1 , wherein analyzing the second image frame using the second process comprises analyzing the second image frame using an iterative detection process.
7. The method as recited in claim 6 , wherein analyzing the second image frame using the iterative detection process comprises:
receiving one or more signals associated with a mobile computing device; and
predicting, based on the received one or more signals, the updated boundary of the displayed document.
8. The method as recited in claim 7 , wherein receiving one or more signals associated with the mobile computing device comprises receiving at least one signal indicating movement of the mobile computing device.
9. The method as recited in claim 7 , wherein predicting the updated boundary of the displayed document comprises:
determining a movement of the mobile computing device based on the one or more signals;
modifying a location of the boundary of the display document in the first image frame based on the determined movement of the mobile computing device; and
applying the modified location to the second image frame.
10. A mobile computing device comprising:
at least one processor; and
at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by the at least one processor, cause the mobile computing device to:
receive an image feed comprising a displayed document;
analyze a first image frame from the image feed using a first process to determine a boundary of the displayed document;
provide, for presentation on a display of the mobile computing device, the first image frame with a document boundary indicator corresponding to the boundary of the displayed document;
analyze, based on the boundary of the displayed document in the first image frame and in parallel with the first process, a second image frame from the image feed using a second process to determine an updated boundary of the displayed document; and
provide, for presentation on the display of the mobile computing device, the second image frame with an updated document boundary indicator corresponding to the updated boundary of the displayed document overlaid on the second image frame.
11. The mobile computing device as recited in claim 10 , wherein receiving the image feed comprises receiving the image feed from a camera associated with the mobile computing device.
12. The mobile computing device as recited in claim 11 , wherein analyzing the first image frame using the first process comprises analyzing the first image frame using a robust detection process comprising:
generating an edge map comprising edges within the first image frame;
identifying, based the edge map, a plurality of edges and a plurality of corners within the first image frame; and
determining one or more quadrilaterals formed from the plurality of edges and the plurality of corners.
13. The mobile computing device as recited in claim 12 , wherein the non-transitory computer-readable storage medium further stores instructions that, when executed by the at least one processor, cause the mobile computing device to:
analyze the one or more quadrilaterals to assign a score to each of the one or more quadrilaterals; and
wherein determining the boundary of the displayed document is based on identifying a quadrilateral from the one or more quadrilaterals that is associated with the highest score.
14. The mobile computing device as recited in claim 13 , wherein analyzing the second image frame using the second process comprises analyzing the second image frame using an iterative detection process.
15. The mobile computing device as recited in claim 14 , wherein analyzing the second image frame using the iterative detection process comprises receiving one or more signals indicating movement of the mobile computing device.
16. The mobile computing device as recited in claim 15 , further comprising generating a predicted location of the displayed document within the second image frame based on the received one or more signals.
17. The mobile computing device as recited in claim 16 , wherein determining the updated boundary of the displayed document within the second image frame is based in part on the predicted location of the displayed document within the second image frame.
18. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computer system to:
receive an image feed comprising a displayed document;
analyze a first image frame from the image feed using a first process to determine a boundary of the displayed document;
provide, for presentation on a display of a client computing device, the first image frame with a document boundary indicator based on the boundary of the displayed document;
analyze, based on the boundary of the displayed document in the first image frame and in parallel with the first process, a second image frame from the image feed using a second process to determine an updated boundary of the displayed document; and
provide, for presentation on the display of the client computing device, the second image frame with an updated document boundary indicator based on the updated boundary of the displayed document.
19. The non-transitory computer-readable medium as recited in claim 18 , wherein:
providing the document boundary indicator for presentation on the display of the client computing device comprises overlaying the document boundary indicator on the first image frame; and
providing the updated document boundary indicator for presentation on the display of the client computing device comprises overlaying the updated document boundary indicator on the second image frame.
20. The non-transitory computer-readable medium as recited in claim 18 , wherein:
analyzing the first image frame using the first process comprises performing a robust displayed document search within the entire first image frame to identify the boundary of the displayed document; and
analyzing the second image frame using the second process comprises performing an iterative detection process to predict an updated location of the displayed document within the second image frame to identify the updated boundary of the display document.