IP Library › Granted Patent US 11,928,558
Granted Patent B1
US 11,928,558 · App. 16/699,247 · Granted Mar 12, 2024

Providing content reviews based on AI/ML output

Inventors: Siddharth Vivek Joshi (Seattle, WA); Anuj Gupta (Seattle, WA); Mark Chien (Bellevue, WA); Jonathan Thomas Greenlee (Bothell, WA); Stefano Stefani (Issaquah, WA); Warren Barkley (Kirkland, WA); Jon I. Turow (Seattle, WA); Sindhu Chejerla (Seattle, WA); Kriti Bharti (Seattle, WA); Prateek Sharma (Seattle, WA)
Assignee: Amazon Technologies, Inc.
G06N20/00G06F16/211
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,928,558
App. No.
16/699,247
Granted
Mar 12, 2024
Kind
B1
Abstract

A request is received associated with a review. Within first content, a first field of interest and a second field of interest are identified and within second content, a third field of interest and a fourth field of interest are identified. A review is generated that includes a first indication of the first field of interest and a second indication of the second field of interest within the first content, as well as a third indication of the third field of interest and a fourth indication of the fourth field of interest within the second content. The review is transmitted to a device of a reviewer for reviewing the content.

Claims (94)

1. A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:

receiving, from a user, a request for reviewing content including a first document and a second document, the request including:

a field of interest to search for within the content; and

a confidence threshold associated with the field of interest;

determining, within the first document and based at least in part on a machine learning (ML) model, a first key and a first value of a first predicted key value pair, the first predicted key value pair corresponding to the field of interest;

determining, within the second document and based at least in part on the ML model, a second key and a second value of a second predicted key value pair, the second predicted key value pair corresponding to the field of interest;

determining that the first key and the first value of the first predicted key value pair correspond to a first key value pair based at least in part on the first key and the first value satisfying the confidence threshold;

determining that the second key and the second value of the second predicted key value pair do not correspond to a second key value pair based at least in part on the second key and the second value not satisfying the confidence threshold; and

generating a review of the content, the review including the second document and the second key and the second value of the second predicted key value pair.

2. The system of claim 1 , the acts further comprising transmitting the review to a device of a reviewer, wherein the device of the reviewer is configured to display the second key and the second value in association with the second document.

3. The system of claim 2 , wherein:

the second key is identified on the device of the reviewer using at least one of a first box, a first outline, or a first color; and

the second value is identified on the device of the reviewer using at least one of a second box, a second outline, or a second color.

4. The system of claim 1 , the acts further comprising:

determining a first confidence associated with the first key corresponding to the field of interest;

determining a second confidence associated with the first value corresponding to the first key;

determining a third confidence associated with the second key corresponding to the field of interest; and

determining a fourth confidence associated with the second value corresponding to the second key, and

wherein:

determining that the first key and the first value satisfy the confidence threshold comprises determining that the first confidence and the second confidence satisfy the confidence threshold; and

determining that the second key and the second value do not satisfy the confidence threshold comprises determining that the third confidence and the fourth confidence do not satisfy the confidence threshold.

5. The system of claim 1 , wherein the content further comprises a third document, the acts further comprising:

determining, within the third document and based at least in part on the ML model, a third key and a third value of a third predicted key value pair, the third predicted key value pair corresponding to the field of interest; and

determining that the third key and the third value of the third predicted key value pair do not correspond to a third key value pair based at least in part on the third key and the third value not satisfying the confidence threshold, and

wherein generating the review of the content includes generating the review including the third document and the third key and the third value of the third predicted key value pair.

6. A method comprising:

receiving a request associated with a review;

determining, based at least in part on receiving the request, first content for review, the first content including a first field of interest and a second field of interest;

determining, based at least in part on receiving the request, second content for review, the second content including a third field of interest and a fourth field of interest;

determining the first content for review and the second content for review based at least in part on the first content and the second content failing to satisfy a confidence threshold;

generating the review, the review including:

a first indication of the first field of interest and a second indication of the second field of interest within the first content; and

a third indication of the third field of interest and a fourth indication of the fourth field of interest within the second content; and

transmitting the review to a device of a reviewer, wherein the device is configured to display:

the first field of interest and the second field of interest within the first content; and

the third field of interest and the fourth field of interest within the second content.

7. The method of claim 6 , wherein:

the first indication is different than the second indication; and

the third indication is different than the fourth indication.

8. The method of claim 6 , wherein:

the request includes one or more conditions associated with determining content for review; and

determining the first content for review and the second content for review is based at least in part on the first content and the second content not satisfying the one or more conditions.

9. The method of claim 6 , further comprising:

determining, using a first machine learning (ML) model, the first field of interest and the second field of interest within the first content;

determining, using a second ML model, one or more first words associated with the first field of interest and one or more second words associated with the second field of interest;

determining, using the first ML model, the third field of interest and the fourth field of interest within the second content; and

determining, using the second ML model, one or more third words associated with the third field of interest and one or more fourth words associated with the fourth field of interest.

10. The method of claim 6 , further comprising:

determining, using a first machine learning (ML) model, the first field of interest and the second field of interest within the first content;

determining, using a second ML model, one or more first objects within a first bounding box associated with the first field of interest and one or more second objects within a second bounding box associated with the second field of interest;

determining, using the first ML model, the third field of interest and the fourth field of interest within the second content; and

determining, using the second ML model, one or more third objects within a third bounding box associated with the third field of interest and one or more fourth objects within a fourth bounding box associated with the fourth field of interest.

11. The method of claim 6 , further comprising determining, for the review, one or more prompts for the reviewer, the one or more prompts including at least one of:

prompting the reviewer to verify at least one of the first field of interest, the second field of interest, the third field of interest, or the fourth field of interest; or

prompting the reviewer to adjust a label associated with at least one of the first field of interest, the second field of interest, the third field of interest, or the fourth field of interest.

12. The method of claim 6 , wherein the device is further configured to display:

a fifth indication associated with the reviewer confirming or denying the first field of interest and the second field of interest being associated with a first key value pair; and

a sixth indication associated with the reviewer confirming or denying the third field of interest and the fourth field of interest being associated with a second key value pair.

13. The method of claim 6 , further comprising:

receiving a second request associated with a second review of third content, the third content including a fifth field of interest and a sixth field of interest;

determining that the fifth field of interest and the sixth field of interest satisfy the confidence threshold; and

refraining from generating the second review of the third content.

14. The method of claim 6 , wherein:

determining that the first content fails to satisfy the confidence threshold includes determining that a first value between the first field of interest and the second field fails to satisfy the confidence threshold; and

determining that the second content fails to satisfy the confidence threshold includes determining that a second value between the third field of interest and the fourth field fails to satisfy the confidence threshold.

15. A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:

determining, within first content and based at least in part on a machine learning (ML) model, a first field of interest and a second field of interest for review;

determining, within second content and based at least in part on the ML model, a third field of interest and a fourth field of interest for review;

based at least in part on the first field of interest, the second field of interest, the third field of interest, and the fourth field of interest, generating a review associated with reviewing the first content and the second content, wherein the review indicates:

the first field of interest and the second field of interest within the first content; and

the third field of interest and the fourth field of interest within the second content; and

transmitting, the review to a device of a reviewer, wherein the device is configured to display:

the first field of interest and the second field of interest within the first content; and

the third field of interest and the fourth field of interest within the second content.

16. The system of claim 15 , wherein:

the first field of interest and the third field of interest are configured to be displayed in association with a first box, a first highlight, or a first outline; and

the second field of interest and the fourth field of interest are configured to be displayed in association with a second box, a second highlight, or a second outline.

17. The system of claim 15 , the acts further comprising:

receiving one or more conditions associated with reviewing the first content and the second content; and

determining the first field of interest, the second field of interest, the third field of interest, and the fourth field of interest based at least in part on the one or more conditions.

18. The system of claim 17 , wherein the one or more conditions comprise a confidence threshold, the acts further comprising:

determining a first confidence corresponding to a first association between the first field of interest and the second field of interest is less than the confidence threshold; and

determining a second confidence corresponding to a second association between the third field of interest and the fourth field of interest is less than the confidence threshold.

19. The system of claim 18 , the acts further comprising:

determining a fifth field of interest within the first content;

determining that a third confidence associated with the fifth field of interest is greater than the confidence threshold; and

determining to not include the fifth field of interest within the review.

20. The system of claim 15 , the acts further comprising receiving a request associated with locating key value pairs within the first content and the second content, wherein:

the first field of interest and the second field of interest are a first predicted key value pair; and

the third field of interest and the fourth field of interest are a second predicted key value pair.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 9, 2024
From: JOSHI, SIDDHARTH VIVEK; GUPTA, ANUJ; CHIEN, MARK; GREENLEE, JONATHAN THOMAS; STEFANI, STEFANO; BARKLEY, WARREN; TUROW, JON I.; CHEJERLA, SINDHU; BHARTI, KRITI; SHARMA, PRATEEK
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 066242/0705 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 20, 2020
From: JOSHI, SIDDHARTH VIVEK; GUPTA, ANUJ; CHIEN, MARK; GREENLEE, JONATHAN THOMAS; STEFANI, STEFANO; BARKLEY, WARREN; TUROW, JON I.; CHEJERLA, SINDHU; BHARTI, KRITI
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 052446/0768 →
Cited By (2)
US 12,591,795 US 12,704,464