IP Library Granted Patent US 11,302,116
Granted Patent B2
US 11,302,116 · App. 16/803,419 · Granted Apr 12, 2022

Device interface extraction

Inventors: William Glaser (Berkeley, CA); Brian Van Osdol (Piedmont, CA)
Assignee: Grabango Co.
G06V40/20G06K9/6232G06V10/245G06V10/40G06V20/00G06V20/52G06V40/10G08B13/196H04L12/282H04L12/2809H04L12/2814H04L67/12H04N5/2252H04N5/2257H04N7/18H04N7/183H05B47/105H05B47/125H05B47/19G06V20/63
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,302,116
App. No.
16/803,419
Granted
Apr 12, 2022
Kind
B2
Abstract

A system and method for visually automated interface integration that includes collecting image data; detecting a device interface source in the image data; processing the image data associated with the device interface source into an extracted interface representation; and exposing at least one access interface to the extracted interface representation.

Claims (41)

1. A method for uncoupled remote device monitoring comprising:

collecting image data;

detecting a device interface source present in the field of view of the image data;

extracting a parameterized representation from image data of the device interface source;

detecting device state;

characterizing an extracted interface representation from the parameterized representation, the characterization being a process based on the device state; and

exposing at least one access interface to the extracted interface representation.

2. The method of claim 1 , wherein detecting device state comprises detecting a displayed mode indicator on the device interface source.

3. The method of claim 1 , wherein detecting device state further comprises detecting state of physical elements of the device interface source.

4. The method of claim 1 , wherein detecting device state comprises detecting an active application of the device interface source.

5. The method of claim 1 , wherein detecting device state comprises detecting audio indicating state of the device interface source.

6. The method of claim 1 , further comprising detecting a user-object in the image data and tracking machine state changes in association with the user-object.

7. The method of claim 1 , further comprising analyzing data patterns of the extracted interface representation.

8. The method of claim 1 , collecting image data comprises collecting audio-visual data and wherein characterizing the extracted interface representation from the parameterized representation further comprises detecting audio output of the device interface source and characterizing the extracted interface representation in part from the detected audio output.

9. The method of claim 1 , wherein characterizing the extracted interface representation from the parameterized representation comprises characterizing at least two sets of information data, used in the extracted interface representation, from the parameterized representation.

10. The method of claim 1 , wherein extracting the parameterized representation of the image data of the device interface source comprises performing character recognition in at least one region and parameterizing graphical state of a second region of the image data.

11. The method of claim 1 , wherein exposing at least one access interface to the extracted interface representation comprises streaming a rendered interface representation to a remote monitoring system, and further comprising selectively elevating priority of the rendered interface representation according to informational data in the rendered interface representation.

12. The method of claim 1 , exposing at least one access interface to the extracted interface representation comprises exposing an application programming interface to a historical data record of information data from the extracted interface representation.

13. The method of claim 1 , wherein the device interface source is a scale, and the extracted interface representation includes measured weight.

14. The method of claim 1 , wherein device state is conditional on detected user interactions with the device interface source.

15. The method of claim 1 , further comprising:

detecting a second device interface source present in the field of view of the image data;

processing the image data associated with the second device interface source into a second extracted interface representation; and

wherein the second extracted interface representation is exposed through the at least one access interface.

16. The method of claim 1 , further comprising monitoring the capture status of a device interface source and generating feedback in response to a change of capture status of the device interface source.

17. The method of claim 16 , further comprising tracking user presence in the field of view of the image data and wherein generating feedback in response to a change of the capture status of the device interface source comprises announcing a change in capture status during user exit of the field of view.

18. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a computing platform, cause the computing platform to:

collecting image data;

detecting a device interface source present in the field of view of the image data;

extracting a parameterized representation from image data of the device interface source;

detecting device state;

characterizing an extracted interface representation from the parameterized representation, the characterization being a process based on the device state; and

exposing at least one access interface to the extracted interface representation.

19. A system comprising of:

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause a computing platform to perform operations comprising:

collecting image data;

detecting a device interface source present in the field of view of the image data;

extracting a parameterized representation from image data of the device interface source;

detecting device state;

characterizing an extracted interface representation from the parameterized representation, the characterization being a process based on the device state; and

exposing at least one access interface to the extracted interface representation.

Assignments (4)
SECURITY INTEREST Recorded Sep 9, 2024
From: GRABANGO CO.
To: GLICKBERG, DANIEL
Reel/Frame 068901/0799 →
SECURITY INTEREST Recorded Jun 5, 2024
From: GRABANGO CO.
To: FIRST-CITIZEN BANK & TRUST COMPANY
Reel/Frame 067637/0231 →
SECURITY INTEREST Recorded Jun 4, 2024
From: GRABANGO CO.
To: GLICKBERG, DANIEL
Reel/Frame 067611/0150 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 5, 2020
From: GLASER, WILLIAM; VAN OSDOL, BRIAN
To: GRABANGO CO.
Reel/Frame 053412/0214 →
Continuity (4)
Continuation 15644799 · Jul 9, 2017
Provisional Application 62360366 · Jul 9, 2016
Provisional Application 62360369 · Jul 9, 2016
Related Publication 20200204391A1 · Jun 25, 2020