Systems and methods for developing and verifying image processing standards for mobile deposit
Systems and methods are provided for assessing whether mobile deposit processing engines meet specified standards for mobile deposit of financial documents. A mobile deposit processing engine (MDE) is evaluated to determine if it can perform technical capabilities for improving the quality of and extracting content from an image of a financial document. A verification process then begins, where the MDE performs the image quality enhancements and text extraction steps on sets of images from a test deck. The results of the processing of the test deck are then evaluated by comparing confidence levels with thresholds to determine if each set of images should be accepted or rejected. Further analysis determines whether any of the sets of images were falsely accepted or rejected in error. An overall error rate is then compared with minimum accuracy criteria, and if the criteria are met, the MDE meets the standard for mobile deposit.
1. A method comprising benchmarking a plurality of mobile document processing engines, which are each different from each other, by, for each of the plurality of mobile document processing engines:
by the mobile document processing engine, for each of a plurality of test images in a test deck representing documents captured by a mobile device,
performing an image processing transaction on the test image, wherein the image processing transaction comprises detecting a magnetic ink character (MICR) line in the test image, and detecting a payment amount in the test image, and
producing a result comprising either an acceptance or rejection of the test image based on the image processing transaction; and,
by a verification server,
comparing the results produced by the mobile document processing engine to known results for the test deck to calculate one or more error rates for the mobile document processing engine, wherein the result produced by the mobile document processing engine for each of the plurality of test images in the test deck matches the known result for that test image when at least the mobile document processing engine successfully performed the image processing transaction for that test image, correctly detected the MICR line in the test image, and correctly detected the payment amount in the test image,
comparing the one or more error rates with one or more accuracy criteria, and
either verifying or rejecting the mobile document processing engine based on the comparison of the one or more error rates with the one or more accuracy criteria.
2. The method of claim 1 , wherein the verification server executes each of the plurality of mobile document processing engines.
3. The method of claim 2 , further comprising, by the verification server, obtaining the plurality of test images and the one or more accuracy criteria from a standards database.
4. The method of claim 1 , further comprising, by the verification server, storing a result of the verification or rejection of the mobile document processing engine in a test results database.
5. The method of claim 1 , wherein the image processing transaction comprises performing one or more mobile image quality assurance (IQA) tests on the test image.
6. The method of claim 5 , wherein the image processing transaction is accepted if a result of the IQA tests meets a defined image quality threshold.
7. The method of claim 1 , wherein the image processing transaction comprises correcting geometric distortions in the test image.
8. The method of claim 1 , wherein the image processing transaction comprises testing compliance with Check 21 standards.
9. The method of claim 1 , wherein the image processing transaction comprises detecting a type of payment represented by the document in the test image.
10. The method of claim 1 , wherein the image processing transaction comprises detecting an endorsement in the test image.
11. The method of claim 1 , wherein the image processing transaction comprises detecting a signature in the test image.
12. The method of claim 1 , wherein the image processing transaction comprises binarizing the test image.
13. The method of claim 1 , wherein a first subset of the plurality of test images are images which should be accepted according to a predetermined standard, and a second subset of the plurality of test images are images which should be rejected according to the predetermined standard.
14. The method of claim 1 , wherein the one or more accuracy criteria comprise a maximum threshold for one or both of false acceptances or false rejections.
15. The method of claim 14 , wherein the one or more accuracy criteria are a plurality of accuracy criteria, and wherein the mobile document processing engine is verified only if all of the plurality of accuracy criteria are met by the one or more error rates.
16. The method of claim 1 , wherein producing a result, comprising either an acceptance of rejection of the test image based on the image processing transaction, comprises accepting the test image when a confidence level of the image processing transaction satisfies a threshold, and rejecting the test image when the confidence level of the image processing transaction does not satisfy the threshold.
17. The method of claim 1 , wherein the plurality of test images comprises pairs of test images, and wherein each pair of test images comprises a first image of a front side of a document and a second image of a back side of the document.
18. A system comprising:
at least one hardware processor; and
memory storing software configured to, when executed by the at least one hardware processor, benchmark a plurality of mobile document processing engines, which are each different from each other, by, for each of the plurality of mobile document processing engines,
executing the mobile document processing engine to, for each of a plurality of test images in a test deck representing documents captured by a mobile device,
perform an image processing transaction on the test image, wherein the image processing transaction comprises detecting a magnetic ink character (MICR) line in the test image, and detecting a payment amount in the test image, and
produce a result comprising either an acceptance or rejection of the test image based on the image processing transaction,
comparing the results produced by the mobile document processing engine to known results for the test deck to calculate one or more error rates for the mobile document processing engine, wherein the result produced by the mobile document processing engine for each of the plurality of test images in the test deck matches the known result for that test image when at least the mobile document processing engine successfully performed the image processing transaction for that test image, correctly detected the MICR line in the test image, and correctly detected the payment amount in the test image,
comparing the one or more error rates with one or more accuracy criteria, and
either verifying or rejecting the mobile document processing engine based on the comparison of the one or more error rates with the one or more accuracy criteria.
19. The method of claim 1 ,
wherein the one or more error rates are a plurality of error rates,
wherein the one or more accuracy criteria are a plurality of accuracy criteria,
wherein each of the plurality of accuracy criteria comprises a threshold for a respective one of the plurality of error rates,
wherein the mobile document processing engine is verified only if all of the plurality of thresholds are satisfied by the plurality of error rates, and
wherein the plurality of error rates comprises false acceptance due to image quality, false acceptance due to the MICR line, false acceptance due to the payment amount, false rejection due to the image quality, false rejection due to the payment amount, and false rejection due to image quality assurance (IQA) test.