IP Library › Granted Patent US 12,475,300
Granted Patent B2
US 12,475,300 · App. 18/744,349 · Granted Nov 18, 2025

Live document detection in a captured video stream

Inventors: Nils Peter Welinder (San Francisco, CA); Peter N. Belhumeur (San Francisco, CA); Ying Xiong (Sunnyvale, CA); Jongmin Baek (San Jose, CA); Simon Kozlov (San Francisco, CA); Thomas Berg (San Francisco, CA); David J. Kriegman (San Diego, CA)
Assignee: Dropbox, Inc.
G06F40/123G06F16/93G06F40/106G06F40/166G06F40/197G06T5/90G06T7/194G06T11/001G06V10/10G06V10/44G06V20/46G06V30/40G06V30/413G06V30/414G06F3/04842G06F3/126G06F40/103G06F40/169G06T5/70G06T11/60G06T2207/20056G06T2207/20081G06T2207/20084G06T2207/30176G06T2210/22G06V10/242G06V10/30G06V30/418
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,475,300
App. No.
18/744,349
Granted
Nov 18, 2025
Kind
B2
Abstract

The present disclosure is directed toward systems and methods to quickly and accurately identify boundaries of a displayed document in a live camera image feed, and provide a document boundary indicator within the live camera image feed. For example, systems and methods described herein utilize different display document detection processes in parallel to generate and provide a document boundary indicator that accurately corresponds with a displayed document within a live camera image feed. Thus, a user of the mobile computing device can easily see whether the document identification system has correctly identified the displayed document within the camera viewfinder feed.

Claims (48)

1 . A computer-implemented method comprising:

detecting a visual representation of a physical document within a live camera feed from a camera on a mobile device;

providing, for display within a graphical user interface of the mobile device, a live boundary indicator associated with the visual representation of the physical document within the live camera feed, the live boundary indicator updating based on changes to a position or an orientation of the visual representation of the physical document within the live camera feed;

capturing an image of the visual representation of the physical document from the live camera feed;

converting the image of the visual representation of the physical document that is in a picture file format to a document file that is in a document file format; and

sending the document file to a second computing device for storage.

2 . The computer-implemented method of claim 1 , wherein sending the document file to the second computing device for storage comprises sending the document file to a content management system causing the content management system to send the document file to the second computing device.

3 . The computer-implemented method of claim 1 , wherein the mobile device and the second computing device are both associated with a user account.

4 . The computer-implemented method of claim 1 , wherein converting the image of the visual representation of the physical document comprises:

generating an enhanced image from the image of the visual representation of the physical document; and

converting the enhanced image of the visual representation of the physical document that is in a picture file format to a document file that is in a document file format

the image of the visual representation of the physical document is in a picture file format; and

converting the image of the visual representation of the physical document to a document file comprises converting the image from the picture file format to a document file format.

5 . The computer-implemented method of claim 1 , wherein the mobile device is associated with a first user account and the second computing device is associated with a second user account.

6 . The computer-implemented method as recited in claim 1 , further comprising providing a shaded overlay comprising a document area within a perimeter of the live boundary indicator and a non-document area outside the perimeter of the live boundary indicator, wherein the document area has a different shading compared to the non-document area.

7 . The computer-implemented method as recited in claim 1 , further comprising:

cropping the image of the visual representation of the physical document based on the live boundary indicator; and

providing a preview of the cropped image prior to converting the image of the visual representation of the physical document to the document file.

8 . A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a mobile device to:

detect a visual representation of a physical document within a live camera feed from a camera on the mobile device, the mobile device associated with a user account;

provide, for display within a graphical user interface of the mobile device, a live boundary indicator associated with the visual representation of the physical document within the live camera feed;

capture an image of the visual representation of the physical document from the live camera feed;

generate an enhanced image from the image of the visual representation of the physical document;

convert the enhanced image of the visual representation of the physical document that is in a picture file format to a document file that is in a document file format; and

provide the document file to a second client device associated with the user account.

9 . The non-transitory computer-readable medium as recited in claim 8 , wherein providing the document file to the second client device associated with the user account comprises providing the document file to a content management system that synchronizes the document file on the second client device.

10 . The non-transitory computer-readable medium as recited in claim 8 , wherein generating the enhanced image from the image of the visual representation of the physical document comprises cropping the image based on the live boundary indicator.

11 . The non-transitory computer-readable medium as recited in claim 8 , wherein:

the image of the visual representation of the physical document is in a picture file format; and

converting the image of the visual representation of the physical document to a document file comprises converting the image from the picture file format to a document file format.

12 . The non-transitory computer-readable medium as recited in claim 8 , wherein providing the document file to the second client device associated with the user account is in response to receiving a user selection of a share file option.

13 . The non-transitory computer-readable medium as recited in claim 8 , further comprising instructions that, when executed by the at least one processor, cause the mobile device to provide a preview of the enhanced image prior to converting the enhanced image to the document file.

14 . The non-transitory computer-readable medium as recited in claim 8 , further comprising instructions that, when executed by the at least one processor, cause the mobile device to provide a shaded overlay comprising a document area within a perimeter of the live boundary indicator and a non-document area outside the perimeter of the live boundary indicator, wherein the document area has a different shading compared to the non-document area.

15 . A system comprising:

at least one processor; and

at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by the at least one processor, cause the system to:

detect a visual representation of a physical document within a live camera feed from a camera on a mobile device, the mobile device associated with a user account;

provide, for display within a graphical user interface of the mobile device, a live boundary indicator associated with the visual representation of the physical document within the live camera feed;

capture an image of the visual representation of the physical document from the live camera feed;

convert the image of the visual representation of the physical document that is in a picture file format to a document file that is in a document file format; and

provide the document file to a second client device associated with the user account.

16 . The system as recited in claim 15 , wherein providing the document file to the second client device associated with the user account comprises providing the document file to a content management system that synchronizes the document file on the second client device.

17 . The system as recited in claim 15 , further comprising instructions that, when executed by the at least one processor, cause the system to:

crop the image of the visual representation of the physical document based on the live boundary indicator; and

provide a preview of the cropped image prior to converting the image of the visual representation of the physical document to the document file.

18 . The system as recited in claim 15 , further comprising instructions that, when executed by the at least one processor, cause the system to provide a shaded overlay comprising a document area within a perimeter of the live boundary indicator and a non-document area outside the perimeter of the live boundary indicator, wherein the document area has a different shading compared to the non-document area.

19 . The system as recited in claim 15 , wherein converting the image of the visual representation of the physical document to a document file comprises converting the image from a picture file format to a document file format.

20 . The system as recited in claim 15 , further comprising instructions that, when executed by the at least one processor, cause the system to adjust a position of the live boundary indicator within the live camera feed based on detecting a change in a position or an orientation of the visual representation of the physical document.

Assignments (2)
SECURITY INTEREST Recorded Dec 12, 2024
From: DROPBOX, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT
Reel/Frame 069604/0611 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 14, 2024
From: WELINDER, NILS PETER; BELHUMEUR, PETER N.; XIONG, YING; BAEK, JONGMIN; KOZLOV, SIMON; BERG, THOMAS; KRIEGMAN, DAVID J.
To: DROPBOX, INC.
Reel/Frame 067736/0171 →
Continuity (6)
Continuation 18295043 · Apr 3, 2023
Continuation 17315027 · May 7, 2021
Continuation 16457423 · Jun 28, 2019
Continuation 15658291 · Jul 24, 2017
Provisional Application 62365914 · Jul 22, 2016
Related Publication 20240330568A1 · Oct 3, 2024
References Cited (62)
US 6671684B1 · Hull et al. · 2003 [cited by applicant]
US 8832549B2 · Mansfield et al. · 2014 [cited by applicant]
US 9105011B2 · Calman et al. · 2015 [cited by applicant]
US 9280691B2 · Kleinman et al. · 2016 [cited by applicant]
US 9305402B2 · Bilbrey et al. · 2016 [cited by applicant]
US 20050100209A1 · Lewis et al. · 2005 [cited by applicant]
US 20060039629A1 · Li et al. · 2006 [cited by applicant]
US 20100060915A1 · Suzuki et al. · 2010 [cited by applicant]
US 20110145068A1 · King et al. · 2011 [cited by applicant]
US 20120238254A1 · Yankovich et al. · 2012 [cited by applicant]
US 20120243048A1 · Asai · 2012 [cited by examiner]
US 20130083003A1 · Perez et al. · 2013 [cited by applicant]
US 20130113943A1 · Wormald et al. · 2013 [cited by applicant]
US 20130173532A1 · Xie · 2013 [cited by examiner]
US 20130182002A1 · Macciola et al. · 2013 [cited by applicant]
US 20130182970A1 · Shustorovich · 2013 [cited by examiner]
US 20140072201A1 · Tilt · 2014 [cited by applicant]
US 20140297256A1 · Rogowski et al. · 2014 [cited by applicant]
US 20150142886A1 · Berry · 2015 [cited by examiner]
US 20150163326A1 · Pan · 2015 [cited by examiner]
US 20150317049A1 · Miura · 2015 [cited by examiner]
US 20160037087A1 · Price · 2016 [cited by applicant]
US 20160071319A1 · Fallon et al. · 2016 [cited by applicant]
US 20160309085A1 · Ilic · 2016 [cited by examiner]
US 20160328607A1 · Krishnan et al. · 2016 [cited by applicant]
US 20160330374A1 · Ilic et al. · 2016 [cited by applicant]
US 20170177194A1 · Lyons · 2017 [cited by examiner]
US 20170235537A1 · Liu · 2017 [cited by examiner]
US 20170257349A1 · Roberts-Hoffman · 2017 [cited by examiner]
US 20180024974A1 · Welinder et al. · 2018 [cited by applicant]
US 20180025251A1 · Welinder · 2018 [cited by examiner]
US 20180184062A1 · Hariri · 2018 [cited by applicant]
Khan Z.H., et al., “Real Time Object Tracking Using BMA,” First International Conference on Information and Communication Technologies, 2005, 5 pages. [cited by applicant]
Kim J., et al., “Video-based Document Tracking: Unifying your Physical and Electronic Desktops,” UIST Proceedings of the Annual ACM symposium on User interface Software and Technology, 2004, 9 pages. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 16/457,423, mailed Aug. 20, 2020, 26 pages. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 16/830,047, mailed Oct. 29, 2020, 24 pages. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 16/830,077, mailed Oct. 29, 2020, 21 pages. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/315,027, mailed Aug. 18, 2022, 17 pages. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/323,838, mailed Oct. 4, 2022, 13 pages. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 18/295,043, mailed Dec. 7, 2023, 9 pages. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 18/295,049, mailed Jan. 19, 2024, 31 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 15/658,291 mailed on Apr. 15, 2019, 8 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 15/658,289, mailed Dec. 11, 2019, 6 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/457,423, mailed Apr. 30, 2021, 2 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/457,423, mailed Feb. 24, 2021, 8 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/457,423, mailed Mar. 26, 2021, 2 pages,. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/830,047, mailed Apr. 29, 2021, 2 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/830,047, mailed Mar. 10, 2021, 6 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/830,077, mailed Apr. 19, 2021, 2 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/830,077, mailed Mar. 22, 2021, 16 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/315,027, mailed Dec. 13, 2022, 08 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/315,027, mailed Dec. 20, 2022, 02 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/323,838, mailed Dec. 2, 2022, 13 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/323,838, mailed Mar. 8, 2023, 02 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 18/295,043, mailed Apr. 1, 2024, 8 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 18/295,043, mailed Apr. 10, 2024, 2 pages. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 18/295,049, mailed May 14, 2024, 6 pages. [cited by applicant]
Office Action for U.S. Appl. No. 15/658,289 mailed on Jan. 28, 2019, 17 pages. [cited by applicant]
Office Action for U.S. Appl. No. 15/658,289 mailed on Jun. 29, 2018, 19 pages. [cited by applicant]
Office Action for U.S. Appl. No. 15/658,291 mailed on Oct. 16, 2018, 13 pages. [cited by applicant]
Office Action for U.S. Appl. No. 15/658,289 mailed on Jun. 25, 2019, 18 pages. [cited by applicant]
Pozik L., et al., “Visual-Inertial Tracking on Android for Augmented Reality applications,” 2012 IEEE Workshop on Environmental Energy and Structural Monitoring Systems, Electronic ISBN: 978-1-4673-2738-1, 7 pages. [cited by applicant]