Document anomaly detection and health analysis
Certain aspects of the disclosure provide systems and methods for analyzing documents, including anomaly detection and health analysis. In some aspects, anomaly detection may further include root cause analysis to determine a foundational anomaly within the document. Health analysis, in some aspects, may further include scoring documents regarding the overall accuracy, completeness, and compliance of the documents. Remedial actions may be taken to reduce anomalies and improve health of documents.
1 . A processing system, comprising: memory comprising computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform steps comprising:
obtaining financial data from a set of financial documents, comprising:
processing each respective financial document of the set of financial documents with natural language processing to generate a respective machine-readable financial document;
anonymizing each respective machine-readable financial document by redacting personally identifiable information within the respective machine-readable financial document; and
converting each respective machine-readable financial document into a structured object comprising the financial data;
extracting a set of financial features from the financial data;
processing the set of financial features with a machine learning model to generate a health score for the set of financial documents, the health score comprising a set of category scores, each category score indicating one of an accuracy of the set of financial documents, a completeness of the set of financial documents, or a compliance of the set of financial documents with a financial area;
detecting an anomaly within the financial data with an anomaly machine learning model;
parsing the anomaly and the financial data with a root cause machine learning model;
determining a root cause of the anomaly within the financial data;
generating, with explainable artificial intelligence (XAI), an explanation of the health score based on processing the set of financial features with the machine learning model, wherein the explanation of the health score indicates a reason for each category score of the set of category scores, and the explanation of the health score indicates the anomaly and the root cause;
outputting, on a user interface, the health score and the explanation of the health score;
providing, on the user interface, at least one financial document of the set of financial documents; and
receiving, via the user interface, from at least one of a plurality of users, an edit to the at least one financial document of the set of financial documents, wherein the edit comprises a remediation of the root cause.
2 . The processing system of claim 1 , wherein:
the structured object comprises a plurality of fields, and
the step further comprises validating each field of the plurality of fields comprising data from the set of financial documents.
3 . The processing system of claim 1 , wherein extracting the set of financial features from the financial data, comprises:
identifying, from the structured object, at least one quantitative field and at least one qualitative field;
contextualizing the at least one quantitative field and the at least one qualitative field based on the financial area associated with the set of financial documents; and
adding the at least one contextualized quantitative field and the at least one contextualized qualitative field to the set of financial features.
4 . The processing system of claim 3 , wherein processing the set of financial features with the machine learning model to generate the health score for the set of financial documents, comprises processing the at least one contextualized quantitative field and the at least one contextualized qualitative field with the machine learning model trained to detect an anomaly between one or more fields of the set of financial features.
5 . The processing system of claim 1 , wherein the steps further comprise reducing a dimensionality of the financial data before feature extraction.
6 . The processing system of claim 1 , wherein the machine learning model is trained on a corpus of financial documents comprising at least one compliant financial document and at least one erroneous financial document.
7 . The processing system of claim 1 , wherein the health score sums the set of category scores.
8 . The processing system of claim 1 , wherein the health score comprises a numerical score and a rating.
9 . The processing system of claim 1 , wherein the explanation of the health score further comprises a recommended action to increase the health score of the set of financial documents.
10 . The processing system of claim 9 , further comprising alerting, on the user interface, to the anomaly and the root cause.
11 . The processing system of claim 9 , wherein outputting, on the user interface, the health score and the explanation of the health score comprises visualizing, on the user interface, a scorecard indicating the health score, the set of category scores, the explanation of the health score, and the recommended action.
12 . The processing system of claim 9 , wherein providing, on the user interface, the anomaly and the root cause, comprises visualizing, on the user interface, a chronological mapping of the financial data, the anomaly, and the root cause, wherein the chronological mapping comprises a heat map or a temporal graph.
13 . A processing system, comprising: memory comprising computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to steps comprising:
obtaining financial data from a set of financial documents, comprising:
processing each respective financial document of the set of financial documents with natural language processing to generate a respective machine-readable financial document;
anonymizing each respective machine-readable financial document by redacting personally identifiable information within the machine-readable financial document; and
converting each respective machine-readable financial document into a structured object comprising the financial data;
extracting a set of financial features from the financial data, comprising:
identifying, from the structured object, at least one quantitative field and at least one qualitative field;
contextualizing the at least one quantitative field and the at least one qualitative field based on the financial area associated with the set of financial documents; and
adding the at least one contextualized quantitative field and the at least one contextualized qualitative field to the set of financial features;
processing the set of financial features with a machine learning model to generate a health score for the set of financial documents, the health score comprising a set of category scores, each category score indicating one of an accuracy of the set of financial documents, a completeness of the set of financial documents, or a compliance of the set of financial documents with a financial area;
detecting an anomaly within the financial data with an anomaly machine learning model;
parsing the anomaly and the financial data with a root cause machine learning model;
determining a root cause of the anomaly within the financial data;
generating, with explainable artificial intelligence (XAI), an explanation of the health score based on processing the set of financial features with the machine learning model, wherein;
the explanation of the health score indicates a reason for each category score of the set of category scores,
the explanation of the health score indicates the anomaly and the root cause, and the explanation of the health score further comprises a recommended action to increase the health score of the set of financial documents; and
outputting, on a user interface, the health score and the explanation of the health score;
providing, on the user interface, at least one financial document of the set of financial documents; and
receiving, via the user interface, from at least one of a plurality of users, an edit to the at least one financial document of the set of financial documents, wherein the edit comprises a remediation of the root cause or the recommended action to increase the health score.
14 . The processing system of claim 13 , wherein:
the structured object comprises a plurality of fields, and
the method further comprises validating each field of the plurality of fields comprising data from the set of financial documents.
15 . The processing system of claim 13 , wherein processing the set of financial features with the machine learning model to generate the health score for the set of financial documents, comprises processing the at least one contextualized quantitative field and the at least one contextualized qualitative field with the machine learning model trained to detect an anomaly between one or more fields of the set of financial features.
16 . The processing system of claim 13 , wherein the steps further comprise reducing a dimensionality of the financial data before feature extraction.
17 . The processing system of claim 13 , wherein the machine learning model is trained on a corpus of financial documents comprising at least one compliant financial document and at least one erroneous financial document.