IP Library Granted Patent US 10,255,253
Granted Patent B2
US 10,255,253 · App. 14/470,491 · Granted Apr 9, 2019

Augmenting and presenting captured data

Inventors: Gregory Akselrod (Seattle, WA); Prashant Thiruvengadachari (Redmond, WA); Eun Ju Nam (Redmond, WA); Zaiqing Nie (Beijing, CN); Yong Cao (Beijing, CN); Pradeep Chilakamarri (Redmond, WA); Bernhard S. J. Kohlmeier (Seattle, WA)
Assignee: Microsoft Technology Licensing, LLC
G06F17/2247G06F17/241G06F17/248
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,255,253
App. No.
14/470,491
Granted
Apr 9, 2019
Kind
B2
Abstract

Captured data can be transformed and augmented for a particular presentation in a document, such as a note of a notebook application, based on an identified entity for the captured data. The particular presentation of captured data can be provided based on entity detection, extraction, and knowledge base resolution and retrieval. Methods, systems, and services are provided that identify a primary entity of an item input to a notebook application and create an entity object for the primary entity of the item at least from one or more structured representations for content associated with the item. A template for presenting the entity object can be determined according to the primary entity, where the template is selected from a set of templates corresponding to different primary entities such that an arrangement and presentation for one primary entity is different than that of another primary entity.

Claims (44)

1. A method of augmenting and presenting captured data, comprising:

receiving one or more structured representations for content associated with an item input to a notebook application;

identifying a primary entity of the item, wherein the primary entity defines the content associated with the item;

creating an entity object for the primary entity of the item from at least the one or more structured representations, wherein the entity object comprises additional information relevant to the item input to the notebook application, the additional information being retrieved from one or more sources other than that of the item;

determining a template for presenting the entity object according to the primary entity, the template being selected from a set of templates, wherein each template from the set of templates corresponds to different primary entities so that an arrangement and presentation for one primary entity is different than that of another primary entity;

providing the entity object in a user interface to the notebook application according to the selected template.

2. The method of claim 1 , wherein the item comprises a uniform resource locator (URL), the method further comprising:

requesting the one or more structured representations for the content associated with the URL from a web data service; and

receiving, from the web data service, one or more models associated with content of a document referenced by the URL, the one or more models providing the one or more structured representations of the item.

3. The method of claim 2 , wherein creating the entity object comprises selecting a single model of the one or more models associated with the content of the document according to a selection criteria.

4. The method of claim 2 , wherein creating the entity object comprises merging attributes of at least two models of the one or more models associated with the content of the document.

5. The method of claim 1 , wherein the primary entity of the object is identified from a minimum number of attributes of the one or more structured representations for the content associated with the item.

6. The method of claim 5 , wherein the primary entity is a recipe; and the minimum number of attributes comprise at least an ingredient array and a steps array.

7. The method of claim 1 , wherein the item comprises an image, the method further comprising:

extracting and classifying information from the image and metadata associated with the image; and

generating at least one of the one or more structured representations by using the extracted and classified information from the image and metadata associated with the image,

wherein identifying the primary entity of the item comprises using the extracted and classified information from the image and metadata associated with the image.

8. The method of claim 7 , further comprising: requesting detailed information corresponding to at least one attribute associated with the primary entity from a knowledgebase or a web data service.

9. The method of claim 7 , wherein the image contains an image of a business card.

10. The method of claim 1 , wherein the item comprises a clip, the method further comprising:

requesting the one or more structured representations for the content associated with the clip from a web data service.

11. A system comprising:

one or more computer readable storage media;

and program instructions stored on the one or more computer readable storage media for presenting content captured for a notebook application that, when executed by a processing system, direct the processing system to:

request an entity object for a primary entity of content associated with an item input to a notebook application service, wherein the primary entity defines the content associated with the item;

determining a template for presenting the entity object according to the primary entity, the template being selected from a set of templates, wherein each template from the set of templates corresponds to different primary entities so that an arrangement and presentation for one primary entity is different than that of another primary entity, and

in a user interface to the notebook application service, surface an entity-specific presentation of the entity object for the content associated with the item input to the notebook application service, wherein an arrangement for the entity-specific presentation of the entity object is controlled by the selected template for the primary entity, wherein the entity object comprises attributes from one or more structured representations for the content associated with the item and additional information relevant to the item, the additional information being retrieved from one or more sources other than that of the item.

12. The system of claim 11 , wherein the item comprises a URL.

13. The system of claim 11 , wherein the item comprises an image.

14. The system of claim 11 , wherein the item comprises a clip.

15. One or more computer readable storage media having program instructions stored thereon for augmenting and presenting captured data, when executed by a processing system, direct the processing system to:

identify a primary entity of an item input to a notebook application, wherein the primary entity defines content associated with the item;

create an entity object for the primary entity of the item at least from one or more structured representations for content associated with the item, wherein the entity object comprises additional information relevant to the item, the additional information being obtained from sources accessible over the Internet or one or more knowledge bases; and

determine a template for presenting the entity object according to the primary entity, the template being selected from a set of templates, wherein each template from the set of templates corresponds to different primary entities so that an arrangement and presentation for one primary entity is different than that of another primary entity;

provide the entity object in a user interface to the notebook application according to the selected template.

16. The media of claim 15 , wherein the item comprises a URL, wherein the program instructions further direct the processing system to:

request the one or more structured representations for the content associated with the URL from a web data service; and

receive, from the web data service, one or more models associated with content of a document referenced by the URL, the one or more models providing the one or more structured representations of the item.

17. The media of claim 16 , wherein the entity object comprises a single model of the one or more models associated with the content of the document selected according to a selection criteria.

18. The media of claim 16 , wherein the entity object comprises merged attributes of at least two models of the one or more models associated with the content of the document.

19. The media of claim 15 , wherein the item comprises an image, wherein the program instructions further direct the processing system to:

extract and classify information from the image and metadata associated with the image; and

generate at least one of the one or more structured representations by using the extracted and classified information from the image and metadata associated with the image,

wherein the primary entity of the item is identified by using the extracted and classified information from the image and metadata associated with the image.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 9, 2015
From: MICROSOFT CORPORATION
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 039025/0454 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 27, 2014
From: AKSELROD, GREGORY; THIRUVENGADACHARI, PRASHANT; NAM, EUN JU; NIE, ZAIQING; CAO, YONG; CHILAKAMARRI, PRADEEP; KOHLMEIER, BERNHARD S.J.
To: MICROSOFT CORPORATION
Reel/Frame 033623/0273 →
Continuity (3)
Continuation PCTCN2014081107 · Jun 30, 2014
Provisional Application 61863241 · Aug 7, 2013
Related Publication 20150046779A1 · Feb 12, 2015
Cited By (1)
US 12,579,160