IP Library › Granted Patent US 11,720,754
Granted Patent B2
US 11,720,754 · App. 17/160,128 · Granted Aug 8, 2023

Systems and methods for extracting evidence to facilitate claim verification

Inventors: Robin Abraham (Redmond, WA); Liang Du (Redmond, WA); Zeeshan Ahmed (Redmond, WA)
Assignee: Microsoft Technology Licensing, LLC
G06F40/30G06F18/214G06F40/284
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,720,754
App. No.
17/160,128
Granted
Aug 8, 2023
Kind
B2
Abstract

Claim verification is facilitated by identifying a selection of textual content within a user interface, accessing a claim detection module, inputting the textual content into the claim detection module to detect one or more claims within the selection of textual content, accessing an evidence extraction module, using the evidence extraction module to automatically search one or more reference repositories for one or more related references that have content that is related to the detected one or more claims, automatically determining whether the content in the one or more related references supports or refutes the one or more claims using a claim verification module, and, in response to determining a set of references of the one or more related references supports or refutes the one or more claims, presenting a support indicator or a refute indicator within the user interface in association with the one or more claims.

Claims (69)

1. A system for facilitating claim verification, comprising:

one or more processors; and

one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to:

identify a selection of textual content within a user interface;

access a claim detection module that has been trained with a first set of ground truth to enable the claim detection module to detect occurrences of claims within text;

input the textual content into the claim detection module to detect one or more claims within the selection of textual content;

in response to detecting the one or more claims within the textual content, access an evidence extraction module that has been trained with a second set of ground truth to enable the evidence extraction module to identify references from one or more reference repositories that have content related to one or more particular types of claims;

further, in response to detecting the one or more claims within the textual content, use the evidence extraction module to automatically search the one or more reference repositories for one or more related references that have one or more pieces of evidence that are determined by the evidence extraction module to be related to the detected one or more claims;

in response to identifying the one or more related references, automatically determine whether the content in the one or more related references supports or refutes the one or more claims using a claim verification module, the claim verification module having been trained with a third set of ground truth to determine whether the one or more pieces of evidence support or refute the one or more claims;

in response to determining a set of references of the one or more related references supports or refutes the one or more claims, present at least one of (i) a support indicator or (ii) a refute indicator within the user interface in association with the one or more claims, the support indicator visually indicating that the one or more claims are supported by evidence, the refute indicator visually indicating that the one or more claims are refuted by evidence;

present a representation of the set of references in the user interface in association with the one or more claims;

present a representation of one or more pieces of evidence of the set of references in the user interface in association with the one or more claims;

receive user input indicating that a reference or a piece of evidence of the set of references is not relevant to the one or more claims; and

further train the evidence extraction module based on the user input indicating that the reference or the piece of evidence of the set of references is not relevant to the one or more claims.

2. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:

in response to determining the set of references of the one or more related references supports the one or more claims, present the support indicator within the user interface in association with the one or more claims.

3. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:

in response to determining the set of references of the one or more related references refutes the one or more claims, present the refute indicator within the user interface in association with the one or more claims.

4. The system of claim 1 , wherein the support indicator or the refute indicator indicates a weight of support or weight of refutation for the one or more claims.

5. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:

present a representation of the set of references in the user interface in association with the one or more claims.

6. The system of claim 5 , wherein the instructions are executable by the one or more processors to further configure the system to:

present a representation of one or more pieces of evidence of the set of references in the user interface in association with the one or more claims.

7. The system of claim 6 , wherein the instructions are executable by the one or more processors to further configure the system to:

receive user input indicating that a reference or a piece of evidence of the set of references is not relevant to the one or more claims; and

refine the evidence extraction module based on the user input indicating that the reference or the piece of evidence of the set of references is not relevant to the one or more claims.

8. The system of claim 6 , wherein the instructions are executable by the one or more processors to further configure the system to:

receive user input indicating that a reference or a piece of evidence of the set of references fails to support the one or more claims; and

refine the claim verification module based on the user input indicating that the reference or the piece of evidence of the set of references fails to support the one or more claims.

9. The system of claim 6 , wherein the instructions are executable by the one or more processors to further configure the system to:

receive user input indicating that a reference or a piece of evidence of the set of references fails to refute the one or more claims; and

refine the claim verification module based on the user input indicating that the reference or the piece of evidence of the set of references fails to refute the one or more claims.

10. The system of claim 1 , wherein the set of references omits a particular reference that fails to satisfy a classification confidence threshold, the classification confidence threshold indicating a level of confidence in a determination by the claim verification module that a reference supports or refutes a claim.

11. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:

present a recommendation for modifying the one or more claims based on the set of references.

12. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:

emphasize one or more portions of the textual content determined by the claim detection module to be contextually associated with the one or more claims;

receive user input modifying the one or more portions of the textual content that are contextually associated with the one or more claims;

redetermine the one or more claims using the claim detection module and based upon the user input modifying the one or more portions of the textual content; and

further train the claim detection module based on the user input modifying the one or more portions of the textual content that are contextually associated with the one or more claims.

13. The system of claim 1 , wherein the one or more related references identified using the evidence extraction module satisfy a threshold of relevance to content of detected claims.

14. A system for facilitating claim verification, comprising:

one or more processors; and

one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to:

identify a selection of textual content within a user interface;

access a claim detection module that has been trained with a first set of ground truth to enable the claim detection module to detect occurrences of claims within text;

input the textual content into the claim detection module to detect one or more claims within the selection of textual content;

in response to detecting the one or more claims within the textual content, access an evidence extraction module that has been trained with a second set of ground truth to enable the evidence extraction module to identify references from one or more reference repositories that have content related to one or more particular types of claims;

further, in response to detecting the one or more claims within the textual content, use the evidence extraction module to automatically search the one or more reference repositories for one or more related references that have one or more pieces of evidence that are determined by the evidence extraction module to be related to the detected one or more claims;

in response to identifying the one or more related references, automatically determine whether the content in the one or more related references supports or refutes the one or more claims using a claim verification module, the claim verification module having been trained with a third set of ground truth to determine whether the one or more pieces of evidence support or refute the one or more claims;

in response to determining a set of references of the one or more related references supports or refutes the one or more claims, present at least one of (i) a support indicator or (ii) a refute indicator within the user interface in association with the one or more claims, the support indicator visually indicating that the one or more claims are supported by evidence, the refute indicator visually indicating that the one or more claims are refuted by evidence;

present a representation of the set of references in the user interface in association with the one or more claims;

present a representation of one or more pieces of evidence of the set of references in the user interface in association with the one or more claims;

receive user input indicating that a reference or a piece of evidence of the set of references fails to support the one or more claims; and

further train the claim verification module based on the user input indicating that the reference or the piece of evidence of the set of references fails to support the one or more claims.

15. A system for facilitating claim verification, comprising:

one or more processors; and

one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to:

identify a selection of textual content within a user interface;

access a claim detection module that has been trained with a first set of ground truth to enable the claim detection module to detect occurrences of claims within text;

input the textual content into the claim detection module to detect one or more claims within the selection of textual content;

in response to detecting the one or more claims within the textual content, access an evidence extraction module that has been trained with a second set of ground truth to enable the evidence extraction module to identify references from one or more reference repositories that have content related to one or more particular types of claims;

further, in response to detecting the one or more claims within the textual content, use the evidence extraction module to automatically search the one or more reference repositories for one or more related references that have one or more pieces of evidence that are determined by the evidence extraction module to be related to the detected one or more claims;

in response to identifying the one or more related references, automatically determine whether the content in the one or more related references supports or refutes the one or more claims using a claim verification module, the claim verification module having been trained with a third set of ground truth to determine whether the one or more pieces of evidence support or refute the one or more claims;

in response to determining a set of references of the one or more related references supports or refutes the one or more claims, present at least one of (i) a support indicator or (ii) a refute indicator within the user interface in association with the one or more claims, the support indicator visually indicating that the one or more claims are supported by evidence, the refute indicator visually indicating that the one or more claims are refuted by evidence;

present a representation of the set of references in the user interface in association with the one or more claims;

present a representation of one or more pieces of evidence of the set of references in the user interface in association with the one or more claims;

receive user input indicating that a reference or a piece of evidence of the set of references fails to refute the one or more claims; and

further train the claim verification module based on the user input indicating that the reference or the piece of evidence of the set of references fails to refute the one or more claims.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 27, 2021
From: ABRAHAM, ROBIN; DU, LIANG; AHMED, ZEESHAN
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 055053/0069 →
Continuity (1)
Related Publication 20220237382A1 · Jul 28, 2022