IP Library Granted Patent US 9,940,367
Granted Patent B1
US 9,940,367 · App. 14/824,553 · Granted Apr 10, 2018

Scoring candidate answer passages

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,940,367
App. No.
14/824,553
Granted
Apr 10, 2018
Kind
B1
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for scoring candidate answer passages. In one aspect, a method includes receiving a query determined to be a question query that seeks an answer response and data identifying resources determined to be responsive to the query; for a subset of the resources: receiving candidate answer passages; determining, for each candidate answer passage, a query term match score that is a measure of similarity of the query terms to the candidate answer passage; determining, for each candidate answer passage, an answer term match score that is a measure of similarity of answer terms to the candidate answer passage; determining, for each candidate answer passage, a query dependent score based on the query term match score and the answer term match score; and generating an answer score that is a based on the query dependent score.

Claims (69)

1. A method performed by data processing apparatus, the method comprising:

receiving a query determined to be a question query that seeks an answer response and data identifying resources determined to be responsive to the query and ordered according to a ranking, the query having query terms;

for each resource in a top-ranked subset of the resources:

receiving candidate answer passages, each candidate answer passage selected from passage units from content of the resource and being eligible to be provided as an answer passage with search results that identify the resources determined to be responsive to the query and being separate and distinct from the search results;

determining, for each candidate answer passage, a query term match score that is a measure of similarity of the query terms to the candidate answer passage;

determining, for each candidate answer passage, an answer term match score that is a measure of similarity of answer terms to the candidate answer passage;

determining, for each candidate answer passage, a query dependent score based on the query term match score and the answer term match score; and

generating an answer score that is a measure of answer quality for the answer response for the candidate answer passage based on the query dependent score.

2. The method of claim 1 , wherein determining a query term match score that is a measure of similarity of the query terms to the candidate answer passage comprises determining a score that is proportional to a number of instances of matches of query terms to terms of the candidate answer passage.

3. The method of claim 1 , wherein determining an answer term match score that is a measure of similarity of answer terms to the candidate answer passage comprises:

generating a list of terms from terms included in the top-ranked subset of resources; and

for each term in the list of terms, generating a term weight for the term.

4. The method of claim 3 , wherein, for each term in the list of terms, generating a term weight for the term comprises:

determining a number of resources in the top-ranked subset of resource in which the term occurs; and

multiplying the number of resources in the top-ranked subset of resource in which the term occurs by an inverse document frequency value for the term.

5. The method of claim 3 , wherein determining an answer term match score that is a measure of similarity of answer terms to the candidate answer passage comprises:

determining a number of times the term occurs in the candidate answer passage; and

multiplying the number of resources the term occurs in the candidate answer passage by the term weight for the term.

6. The method of claim 1 , wherein determining an answer term match score that is a measure of similarity of answer terms to the candidate answer passage comprises:

determining an entity type for an answer response to the question query;

identifying entities described in the candidate answer passage; and

reducing the answer term match score if none of the identified entities match an entity of the entity type determined for the answer response.

7. The method of claim 1 , further comprising:

determining, for each candidate answer passage, a query independent score for the candidate answer passage, wherein the query independent score is independent of the query and query dependent score; and

wherein generating an answer score that is a measure of answer quality for the answer response for the candidate answer passage based on the query dependent score comprises generating the answer score based on the query dependent score and the query independent score.

8. The method of claim 7 , wherein determining the query independent score for the candidate answer passage comprises determining a score for the candidate answer passage based on a ranking score of the resource from which the candidate answer passage was created.

9. The method of claim 7 , wherein determining the query independent score for the candidate answer passage comprises determining a score for the candidate answer passage based on a reputation of the resource from which the candidate answer passage was created.

10. The method of claim 7 , wherein determining the query independent score for the candidate answer passage comprises determining a score for the candidate answer passage based on a site quality score of a site that hosts the resource from which the candidate answer passage was created.

11. The method of claim 7 , wherein determining the query independent score for the candidate answer passage comprises:

determining in the candidate answer passage includes passage units from two different sections of the resource from which it was created; and

reducing the query independent score if the candidate answer passage includes passage units from two different sections of the resource.

12. The method of claim 7 , wherein determining the query independent score for the candidate answer passage comprises:

determining in the candidate answer passage begins with a discourse boundary term; and

reducing the query independent score if the candidate answer passage beings with a discourse boundary term.

13. A system, comprising:

a data processing apparatus; and

a memory storage system in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:

receiving a query determined to be a question query that seeks an answer response and data identifying resources determined to be responsive to the query and ordered according to a ranking, the query having query terms;

for each resource in a top-ranked subset of the resources:

receiving candidate answer passages, each candidate answer passage selected from passage units from content of the resource and being eligible to be provided as an answer passage with search results that identify the resources determined to be responsive to the query and being separate and distinct from the search results;

determining, for each candidate answer passage, a query term match score that is a measure of similarity of the query terms to the candidate answer passage;

determining, for each candidate answer passage, an answer term match score that is a measure of similarity of answer terms to the candidate answer passage;

determining, for each candidate answer passage, a query dependent score based on the query term match score and the answer term match score; and

generating an answer score that is a measure of answer quality for the answer response for the candidate answer passage based on the query dependent score.

14. The system of claim 13 , wherein determining a query term match score that is a measure of similarity of the query terms to the candidate answer passage comprises determining a score that is proportional to a number of instances of matches of query terms to terms of the candidate answer passage.

15. The system of claim 13 , wherein determining an answer term match score that is a measure of similarity of answer terms to the candidate answer passage comprises:

generating a list of terms from terms included in the top-ranked subset of resources; and

for each term in the list of terms, generating a term weight for the term.

16. The system of claim 15 , wherein, for each term in the list of terms, generating a term weight for the term comprises:

determining a number of resources in the top-ranked subset of resource in which the term occurs; and

multiplying the number of resources in the top-ranked subset of resource in which the term occurs by an inverse document frequency value for the term.

17. The system of claim 15 , wherein determining an answer term match score that is a measure of similarity of answer terms to the candidate answer passage comprises:

determining a number of times the term occurs in the candidate answer passage; and

multiplying the number of resources the term occurs in the candidate answer passage by the term weight for the term.

18. The system of claim 13 , wherein determining an answer term match score that is a measure of similarity of answer terms to the candidate answer passage comprises:

determining an entity type for an answer response to the question query;

identifying entities described in the candidate answer passage; and

reducing the answer term match score if none of the identified entities match an entity of the entity type determined for the answer response.

19. The system of claim 13 , the operations further comprising:

determining, for each candidate answer passage, a query independent score for the candidate answer passage, wherein the query independent score is independent of the query and query dependent score; and

wherein generating an answer score that is a measure of answer quality for the answer response for the candidate answer passage based on the query dependent score comprises generating the answer score based on the query dependent score and the query independent score.

20. A memory storage system storing instructions executable by a data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:

receiving a query determined to be a question query that seeks an answer response and data identifying resources determined to be responsive to the query and ordered according to a ranking, the query having query terms;

for each resource in a top-ranked subset of the resources:

receiving candidate answer passages, each candidate answer passage selected from passage units from content of the resource and being eligible to be provided as an answer passage with search results that identify the resources determined to be responsive to the query and being separate and distinct from the search results;

determining, for each candidate answer passage, a query term match score that is a measure of similarity of the query terms to the candidate answer passage;

determining, for each candidate answer passage, an answer term match score that is a measure of similarity of answer terms to the candidate answer passage;

determining, for each candidate answer passage, a query dependent score based on the query term match score and the answer term match score; and

generating an answer score that is a measure of answer quality for the answer response for the candidate answer passage based on the query dependent score.

Assignments (3)
CERTIFICATE OF CONVERSION Recorded Jan 17, 2018
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 045080/0366 →
CHANGE OF NAME Recorded Oct 20, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044567/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 15, 2015
From: BAKER, STEVEN D.; VENKATACHARY, SRINIVASAN; BRENNAN, ROBERT ANDREW; BJORNSSON, PER; LIU, YI; SHEMTOV, HADAR; CIARAMITA, MASSIMILIANO; TSOCHANTARIDIS, IOANNIS
To: GOOGLE INC.
Reel/Frame 036799/0065 →