IP Library Granted Patent US 10,049,476
Granted Patent B1
US 10,049,476 · App. 14/714,228 · Granted Aug 14, 2018

System and method of creating an editable text and images from a captured image of a hand-drawn and/or static two-dimensional diagram

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,049,476
App. No.
14/714,228
Granted
Aug 14, 2018
Kind
B1
Abstract

A method of creating an editable image and editable text from a hand-drawn or other static two-dimensional diagram may include receiving a raw image from an image acquisition device; modifying the raw image to a modified image to reduce noise, normalize raw image data, and reduce pixels; recognizing horizontal, vertical, and diagonal lines in the modified image using a line recognizer; connecting the lines in the modified image to form connected lines using a connector that detects and joins proximally positioned terminal ends of the lines in the modified image; recognizing areas bounded by the connected lines as bounded objects using a bounded object recognizer; and/or identifying and classifying the bounded objects using an object identifier.

Claims (49)

1. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

receive an image from an image acquisition device;

recognize lines in the image as follows:

search for white pixels in the image;

when a white pixel is found:

(a) search surrounding pixels of the white pixel to create a longest straight line passing through only white pixels and mark pixels close to the longest straight line as found;

(b) from the start of the longest straight line:

(i) search the surrounding pixels to find the longest connected straight line passing through only white pixels and marking the pixels close to the longest connected straight line as found;

(ii) continue (i) until there are no more white pixels or found pixels;

(iii) if an intersection is found for forming two connected lines, save the intersection and continue with step (i) in each available direction from the intersection;

(c) from the end of the longest straight line, perform steps (i), (ii), and (iii);

connect the lines from the line recognizer to form connected lines by adding to the image a connector that joins proximally positioned terminal ends of the lines;

recognize bounded objects in the image using a bounded object recognizer as follows:

identify lines that end close to another line and when found connect them by adding a connector to the image; and

(e) identify sets of connected lines that form closed shapes and extract the shapes in order of shortest to largest path; and

on a machine display, overlay the shapes on the image as independently manipulable objects.

2. The computer-readable storage medium of claim 1 , further including instructions that when executed by the computer, cause the computer to:

reduce pixels in the image to a single channel;

apply a smoothing filter to image to generate a filtered image; and

subtract the filtered image from the image.

3. The computer-readable storage medium of claim 1 , further including instructions that when executed by the computer, cause the computer to:

determine a starting threshold based on an intensity gradient of the image;

adjust the starting threshold to minimize a number of isolated pixels in the image;

perform a threshold operation to further reduce the pixels; and

perform dilation/erosion operations to adjust a thickness of linear segments formed by the pixels.

4. The computer-readable storage medium of claim 1 , further including instructions that when executed by the computer, cause the computer to:

scan through strings in the image;

for each string, identify two vertexes that are farthest apart;

calculate a shortest route between the vertexes;

generate a set of points representing a line along the shortest route;

fit a line to the set of points;

if a length of the line is greater than the minimum line length parameter;

calculate an average fit error of the points to the line;

if the fit error is less than the minimum line length parameter;

(i) remove line segments making up the line from the string;

(ii) move all remaining disconnected segments to new strings;

(iii) define a new line from best-fit parameters for the points; and

add the new line to the lines from the line recognizer.

5. The computer-readable storage medium of claim 1 , further including instructions that when executed by the computer, cause the computer to adjust the length of the lines by:

scanning each bounded object and calculating a distance between the two farthest vertices;

calculating an average of the distances; and

adjusting a minimum line length parameter based on the average.

6. The computer-readable storage medium of claim 1 , further comprising instructions that when executed by the computer, cause the computer to identify strings that are below a threshold length and that overlay other strings and move such strings to a chaff category, as follows:

scan through the strings;

calculate a length of each string;

if the string is shorter than a first parameter, move it to chaff; and

if the string is shorter than a second parameter and overlays another string, move it to chaff.

7. The computer-readable storage medium of claim 6 , further comprising instructions that when executed by the computer, cause the computer to identify strings that qualify as chaff, as follows:

check each string against other strings to see if it is (a) isolated from the other strings, and (b) below a threshold length.

Assignments (8)
RELEASE OF SECURITY INTEREST IN SPECIFIED INTELLECTUAL PROPERTY [RF 68892 0761] Recorded Mar 4, 2026
From: WILMINGTON TRUST, NATIONAL ASSOCIATION
To: CLARIVATE ANALYTICS (COMPUMARK) INC.; CLARIVATE ANALYTICS (US) LLC; CLARIVATE ANALYTICS (COMPUMARK) LIMITED; CLARIVATE ANALYTICS (UK) LIMITED; DNSTINATION INC.; DOMAIN FORTRESS INC.; MARKMONITOR CORPORATE SERVICES INC.; MARKMONITOR INC.; MICROPATENT LLC; CAMELOT UK BIDCO LIMITED; DECISION RESOURCES, INC.; DR/DECISION RESOURCES, LLC; CPA GLOBAL (FIP) LLC; INNOGRAPHY, INC.; CPA GLOBAL (FTF) INC.; CPA GLOBAL (LANDON IP) INC.; MASTER DATA CENTER, INC.; PATENT RESOURCES GROUP, INC.; IPFOLIO CORPORATION; OLCOTT INTERNATIONAL & CO., L.L.C.; PROQUEST LLC; EX LIBRIS (USA) INC.; INNOVATIVE INTERFACES INCORPORATED; ROWAN TELS CORP.; GLOBAL QMS, INC.
Reel/Frame 075063/0219 →
PATENT SECURITY AGREEMENT (CA) Recorded Sep 13, 2024
From: ROWAN TELS CORP.
To: BANK OF AMERICA, N.A. AS COLLATERAL AGENT
Reel/Frame 068892/0481 →
PATENT SECURITY AGREEMENT (2021 INDENTURE) Recorded Sep 13, 2024
From: ROWAN TELS CORP.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION AS COLLATERAL AGENT
Reel/Frame 068892/0689 →
PATENT SECURITY AGREEMENT (2019 INDENTURE) Recorded Sep 13, 2024
From: ROWAN TELS CORP.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION AS COLLATERAL AGENT
Reel/Frame 068892/0761 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 7, 2022
From: TURBOPATENT INC.
To: ROWAN TELS CORP.
Reel/Frame 059865/0703 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 19, 2018
From: SNAPPT; JOLT LABS
To: TURBOPATENT,CORP.
Reel/Frame 045588/0038 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 29, 2018
From: DEMBOSKI, BEN; WATTS, PAUL; TAKEE, MASANORI KING; BILLMAIER, JAMES A.; BILLMAIER, DAVID P.; BILLMAIER, KRISTOPHER C.
To: SNAPPT
Reel/Frame 045383/0494 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 26, 2017
From: BILLMAIER, JAMES A.
To: TURBOPATENT, INC.
Reel/Frame 043696/0122 →