Establishing a proficiency baseline for any domain specific natural language processing
A system, method, and computer-readable medium for performing a domain specific evaluation operation comprising: storing domain specific data within a business query repository; determining and understanding variations within language for a domain specific category; performing a test planner operation on an identified NLP system, the test planner operation allowing a user to select a test plan to apply to the identified NLP system; and, evaluating the identified NLP system using a text planner output.
1. A method for performing a domain specific evaluation operation within a customer service interaction estimation environment, the customer service interaction environment comprising a customer service interaction estimation system executing on a hardware processor of an information handling system, the method comprising:
storing domain specific data within a business query repository;
determining and understanding variations within language for a domain specific category;
performing a test planner operation on an identified natural language processing (NLP) system, the test planner operation allowing a user to select a test plan to apply to the identified NLP system, the identified NLP system comprising a domain specific NLP system, the domain specific NLP system being designed to function within a specific domain;
evaluating the identified NLP system using a test planner output, the evaluating establishing a proficiency baseline of the identified NLP system for the specific domain, the specific domain corresponding to the domain specific category, the proficiency baseline indicating how well the identified NLP system performs in the specific domain corresponding to the domain specific category;
training the identified natural language processing system to function within the specific domain based upon the proficiency baseline of the identified NLP system for the specific domain; and,
fabricating a product via a custom product fabrication system, the product being fabricated to include components identified via the identified natural language processing system.
2. The method of claim 1 , wherein:
the determining and understanding of variation is performed using lexical diversity measurements.
3. The method of claim 2 , wherein:
the lexical diversity measurements are associated with the categories of queries the identified NLP system is anticipated to handle.
4. The method of claim 1 , wherein:
the test plan includes a hypothesis testing type test plan, the hypothesis testing type test plan using a statistical p value to indicate whether a test hypothesis can be accepted or not.
5. The method of claim 1 , wherein:
the identified NLP system is designed based upon at least one of specific business intents and specific questions.
6. The method of claim 1 , further comprising:
generating an NLP proficiency value based upon the evaluating, the NLP proficiency value providing an indication of proficiency of the identified NLP system; and,
presenting the NLP proficiency value via a domain specific evaluation user interface, the domain specific evaluation user interface being presented via a user device, the domain specific evaluation user interface providing a visualization the proficiency of the identified NLP system.
7. A system comprising:
a hardware processor;
a data bus coupled to the processor; and
a non-transitory, computer-readable storage medium embodying computer program code for performing a domain specific evaluation operation within a customer service interaction estimation environment, the customer service interaction environment comprising a customer service interaction estimation system executing on the hardware processor of the system, the non-transitory, computer-readable storage medium being coupled to the data bus, the computer program code interacting with a plurality of computer operations and comprising instructions executable by the processor and configured for:
storing domain specific data within a business query repository;
determining and understanding variations within language for a domain specific category;
performing a test planner operation on an identified natural language processing (NLP) system, the test planner operation allowing a user to select a test plan to apply to the identified NLP system, the identified NLP system comprising a domain specific NLP system, the domain specific NLP system being designed to function within a specific domain;
evaluating the identified NLP system using a test planner output, the evaluating establishing a proficiency baseline of the identified NLP system for the specific domain, the specific domain corresponding to the domain specific category, the proficiency baseline indicating how well the identified NLP system performs in the specific domain corresponding to the domain specific category;
training the identified natural language processing system to function within the specific domain based upon the proficiency baseline of the identified NLP system for the specific domain; and,
fabricating a product via a custom product fabrication system, the product being fabricated to include components identified via the identified natural language processing system.
8. The system of claim 7 , wherein:
the determining and understanding of variation is performed using lexical diversity measurements.
9. The system of claim 8 , wherein:
the lexical diversity measurements are associated with the categories of queries the identified NLP system is anticipated to handle.
10. The system of claim 7 , wherein:
the test plan includes a hypothesis testing type test plan, the hypothesis testing type test plan using a statistical p value to indicate whether a test hypothesis can be accepted or not.
11. The system of claim 7 , wherein:
the identified NLP system is designed based upon at least one of specific business intents and specific questions.
12. The system of claim 7 , wherein the instructions are further configured for:
generating an NLP proficiency value based upon the evaluating, the NLP proficiency value providing an indication of proficiency of the identified NLP system; and,
presenting the NLP proficiency value via a domain specific evaluation user interface, the domain specific evaluation user interface being presented via a user device, the domain specific evaluation user interface providing a visualization the proficiency of the identified NLP system.
13. A non-transitory, computer-readable storage medium embodying computer program code for performing a domain specific evaluation operation within a customer service interaction estimation environment, the customer service interaction environment comprising a customer service interaction estimation system executing on a hardware processor of an information handling system, the computer program code comprising computer executable instructions configured for:
storing domain specific data within a business query repository;
determining and understanding variations within language for a domain specific category;
performing a test planner operation on an identified natural language processing (NLP) system, the test planner operation allowing a user to select a test plan to apply to the identified NLP system, the identified NLP system comprising a domain specific NLP system, the domain specific NLP system being designed to function within a specific domain; and,
evaluating the identified NLP system using a test planner output, the evaluating establishing a proficiency baseline of the identified NLP system for the specific domain, the specific domain corresponding to the domain specific category, the proficiency baseline indicating how well the identified NLP system performs in the specific domain corresponding to the domain specific category;
training the identified natural language processing system to function within the specific domain based upon the proficiency baseline of the identified NLP system for the specific domain; and,
fabricating a product via a custom product fabrication system, the product being fabricated to include components identified via the identified natural language processing system.
14. The non-transitory, computer-readable storage medium of claim 13 , wherein:
the determining and understanding of variation is performed using lexical diversity measurements.
15. The non-transitory, computer-readable storage medium of claim 14 , wherein:
the lexical diversity measurements are associated with the categories of queries the identified NLP system is anticipated to handle.
16. The non-transitory, computer-readable storage medium of claim 13 , wherein:
the test plan includes a hypothesis testing type test plan, the hypothesis testing type test plan using a statistical p value to indicate whether a test hypothesis can be accepted or not.
17. The non-transitory, computer-readable storage medium of claim 13 , wherein:
the identified NLP system is designed based upon at least one of specific business intents and specific questions.
18. The non-transitory, computer-readable storage medium of claim 13 , wherein the computer executable instructions are further configured for:
generating an NLP proficiency value based upon the evaluating, the NLP proficiency value providing an indication of proficiency of the identified NLP system; and,
presenting the NLP proficiency value via a domain specific evaluation user interface, the domain specific evaluation user interface being presented via a user device, the domain specific evaluation user interface providing a visualization the proficiency of the identified NLP system.
19. The non-transitory, computer-readable storage medium of claim 13 , wherein the computer executable instructions are deployable to a client system from a server system at a remote location.
20. The non-transitory, computer-readable storage medium of claim 13 , wherein the computer executable instructions are provided by a service provider to a user on an on-demand basis.