Verification of extracted data
View Patent ↗Facts are extracted from speech and recorded in a document using codings. Each coding represents an extracted fact and includes a code and a datum. The code may represent a type of the extracted fact and the datum may represent a value of the extracted fact. The datum in a coding is rendered based on a specified feature of the coding. For example, the datum may be rendered as boldface text to indicate that the coding has been designated as an “allergy.” In this way, the specified feature of the coding (e.g., “allergy”-ness) is used to modify the manner in which the datum is rendered. A user inspects the rendering and provides, based on the rendering, an indication of whether the coding was accurately designated as having the specified feature. A record of the user's indication may be stored, such as within the coding itself.
1. A method performed by at least one computer processor executing computer program instructions tangibly stored on at least one non-transitory computer-readable medium, the method comprising:
(A) identifying a document including a first coding having a first feature encoding a first concept, the first coding being associated with a first code;
(B) rendering the first coding on an output device to have a visual characteristic that is based on the first feature, without rendering the first code;
(C) receiving, via an input device, a first indication from a user of whether the rendering is accurate;
(D) identifying, based on the first indication received from the user, a verification status of the first coding, wherein the verification status of the first coding indicates whether the first code represents the first concept, comprising:
(D)(1) if the first indication indicates that the rendering is accurate, then identifying a verification status of the first coding indicating that the first coding is accurate; and
(D)(2) otherwise, identifying a verification status of the first coding indicating that the first coding is inaccurate;
(E) identifying, based on the verification status, a level of trust that the first coding is correct; and
(F) storing, in the first coding, a non-transitory record of the identified level of trust.
2. The method of claim 1 further comprising:
if the verification status of the first coding indicates that the first coding is inaccurate, then modifying the first feature of the first coding to produce a modified first feature and storing a non-transitory record of the modified first feature.
3. The method of claim 2 , wherein the first feature comprises a specified relationship between the first coding and a second coding, and further comprising modifying the relationship between the first coding and the second coding.
4. The method of claim 3 , wherein the second coding includes the first coding, wherein the first feature comprises inclusion of the first coding in the second coding, and wherein modifying comprises removing the first coding from the second coding.
5. The method of claim 3 , wherein the first and second codings are disjoint and wherein the first feature is represented by a first feature identifier in the document.
6. The method of claim 3 , wherein modifying comprises severing the specified relationship.
7. The method of claim 1 , wherein the first code comprises first text, and wherein (D) comprises identifying a verification status of the first coding, wherein the verification status of the first coding indicates whether the first text describes the first concept.
8. The method of claim 1 , further comprising:
(G) storing, in the document, a record of the verification status of the first coding.
9. The method of claim 8 , wherein (G) comprises storing the record in the first coding.
10. The method of claim 1 , wherein (D) comprises determining whether the first code represents a concept having the first feature.
11. The method of claim 1 , wherein the visual characteristic comprises a text formatting characteristic.
12. The method of claim 11 , wherein the visual characteristic comprises boldface.
13. The method of claim 1 , wherein (E) further comprises identifying a first level of trust that the first coding is correct if the first indication indicates that the rendering is accurate.
14. The method of claim 13 further comprising identifying a second level of trust that a second coding is correct based upon a second indication that the second coding is inaccurate, the second level of trust indicating a lower level of trust than the first level of trust.
15. A non-transitory computer readable medium comprising computer program instructions tangibly stored on the computer readable medium, wherein the computer program instructions are executable by at least one computer processor to perform a method, the method comprising:
(A) identifying a document including a first coding having a first feature encoding a first concept, the first coding being associated with a first code;
(B) rendering the first coding on an output device to have a visual characteristic that is based on the first feature, without rendering the first code;
(C) receiving, via an input device, a first indication from a user of whether the rendering is accurate;
(D) identifying, based on the first indication received from the user, a verification status of the first coding, wherein the verification status of the first coding indicates whether the first code represents the first concept, comprising:
(D)(1) if the first indication indicates that the rendering is accurate, then identifying a verification status of the first coding indicating that the first coding is accurate; and
(D)(2) otherwise, identifying a verification status of the first coding indicating that the first coding is inaccurate;
(E) identifying, based on the verification status, a level of trust that the first coding is correct; and
(F) storing, in the first coding, a non-transitory record of the identified level of trust.
16. The computer readable medium of claim 15 wherein the method further comprises:
if the verification status of the first coding indicates that the first coding is inaccurate, then modifying the first feature of the first coding to produce a modified first feature and storing a non-transitory record of the modified first feature.
17. The computer readable medium of claim 16 , wherein the first feature comprises a specified relationship between the first coding and a second coding, and wherein the method further comprises modifying the relationship between the first coding and the second coding.
18. The computer readable medium of claim 17 , wherein the second coding includes the first coding, wherein the first feature comprises inclusion of the first coding in the second coding, and wherein the method comprises removing the first coding from the second coding.
19. The computer readable medium of claim 17 , wherein the first and second codings are disjoint and wherein the first feature is represented by a first feature identifier in the document.
20. The computer readable medium of claim 17 , wherein modifying comprises severing the specified relationship.
21. The computer readable medium of claim 15 , wherein the first code comprises first text, and wherein (D) comprises identifying a verification status of the first coding, wherein the verification status of the first coding indicates whether the first text describes the first concept.
22. The computer readable medium of claim 15 , further comprising:
(G) storing, in the document, a record of the verification status of the first coding.
23. The computer readable medium of claim 15 , wherein (G) comprises storing the record in the first coding.
24. The computer readable medium of claim 15 , wherein (D) comprises determining whether the first code represents a concept having the first feature.
25. The computer readable medium of claim 15 , wherein the visual characteristic comprises a text formatting characteristic.
26. The computer readable medium of claim 25 , wherein the visual characteristic comprises boldface.
27. The computer readable medium of claim 15 , wherein (E) further comprises identifying a first level of trust that the first coding is correct if the first indication indicates that the rendering is accurate.
28. The computer readable medium of claim 27 further comprising identifying a second level of trust that a second coding is correct based upon a second indication that the second coding is inaccurate, the second level of trust indicating a lower level of trust than the first level of trust.
29. A method performed by at least one computer processor executing computer program instructions tangibly stored on at least one non-transitory computer-readable medium, the method comprising:
(A) identifying a document including a first coding having a first feature encoding a first concept, the first coding being associated with a first code;
(B) rendering the first coding on an output device to have a visual characteristic that is based on the first feature, without rendering the first code;
(C) receiving, via an input device, a first indication from a user of whether the rendering is accurate;
(D) identifying, based on the first indication received from the user, a first level of trust that the first coding is correct;
(E) storing, in the first coding, a non-transitory record of the identified first level of trust;
(F) determining that a second coding is tagged as an automatically derived coding;
(G) identifying a second level of trust that the second coding is correct, the second level of trust indicating a lower level of trust than the first level of trust; and
(H) storing, in the second coding, a non-transitory record of the identified second level of trust.
30. The method of claim 29 further comprising:
if the verification status of the first coding indicates that the first coding is inaccurate, then modifying the first feature of the first coding to produce a modified first feature and storing a non-transitory record of the modified first feature.
31. The method of claim 30 , wherein the first feature comprises a specified relationship between the first coding and a second coding, and further comprising modifying the relationship between the first coding and the second coding.
32. The method of claim 31 , wherein the second coding includes the first coding, wherein the first feature comprises inclusion of the first coding in the second coding, and wherein modifying further comprises removing the first coding from the second coding.
33. The method of claim 31 , wherein the first and second codings are disjoint and wherein the first feature is represented by a first feature identifier in the document.
34. The method of claim 31 , wherein modifying comprises severing the specified relationship.
35. The method of claim 29 , further comprising:
(I) storing, in the document, a record of the verification status of the first coding.
36. The method of claim 35 , wherein (I) comprises storing the record in the first coding.
37. The method of claim 29 , wherein the visual characteristic comprises a text formatting characteristic.
38. The method of claim 37 , wherein the visual characteristic comprises boldface.
39. A non-transitory computer readable medium comprising computer program instructions tangibly stored on the computer readable medium, wherein the computer program instructions are executable by at least one computer processor to perform a method, the method comprising:
(A) identifying a document including a first coding having a first feature encoding a first concept, the first coding being associated with a first code;
(B) rendering the first coding on an output device to have a visual characteristic that is based on the first feature, without rendering the first code;
(C) receiving, via an input device, a first indication from a user of whether the rendering is accurate;
(D) identifying, based on the first indication received from the user, a first level of trust that the first coding is correct;
(E) storing, in the first coding, a non-transitory record of the identified first level of trust;
(F) determining that a second coding is tagged as an automatically derived coding;
(G) identifying a second level of trust that the second coding is correct, the second level of trust indicating a lower level of trust than the first level of trust; and
(H) storing, in the second coding, a non-transitory record of the identified second level of trust.
40. The computer readable medium of claim 39 wherein the method further comprises:
if the verification status of the first coding indicates that the first coding is inaccurate, then modifying the first feature of the first coding to produce a modified first feature and storing a non-transitory record of the modified first feature.
41. The computer readable medium of claim 40 , wherein the first feature comprises a specified relationship between the first coding and a second coding, and wherein the method further comprises modifying the relationship between the first coding and the second coding.
42. The computer readable medium of claim 41 , wherein the second coding includes the first coding, wherein the first feature comprises inclusion of the first coding in the second coding, and wherein modifying further comprises removing the first coding from the second coding.
43. The computer readable medium of claim 41 , wherein the first and second codings are disjoint and wherein the first feature is represented by a first feature identifier in the document.
44. The computer readable medium of claim 41 , wherein modifying comprises severing the specified relationship.
45. The computer readable medium of claim 39 , further comprising:
(I) storing, in the document, a record of the verification status of the first coding.
46. The computer readable medium of claim 45 , wherein (I) comprises storing the record in the first coding.
47. The computer readable medium of claim 39 , wherein the visual characteristic comprises a text formatting characteristic.
48. The computer readable medium of claim 47 , wherein the visual characteristic comprises boldface.