Real-time natural language processing and fulfillment
A system and method of real-time feedback confirmation to solicit a virtual assistant response from an evolving semantic state of at least a portion of an utterance. A user accesses a virtual assistant on an electronic device having the system and/or method configured to capture a command, a question, and/or a fulfillment request from audio such as, the speech emitted from the speaking user. The speech may be intercepted by a speech engine configured to transcribe the speech into text that is matched with the fragment pattern's regular expression to generate a fragment and/or the speech may be processed with a machine learning model to identify fragments. The fragments are identified by a domain handler configured to update a data structure of the current semantic state of the utterance in real-time on an interface of an electronic device.
1. A computer-implemented method comprising:
transcribing audio, using a speech engine, to create a transcription;
executing a fragment identifier on a portion of the transcription, the fragment identifier detecting a fragment and outputting the fragment;
executing a fragment integrator that receives the fragment from the fragment identifier, and matches the fragment against predefined rules; and
responsive to detecting by the fragment identifier and responsive to matching by the fragment integrator, invoking a domain handler with the fragment.
2. The method of claim 1 , further comprising updating a conversation state data structure with entity values from the fragment.
3. The method of claim 1 , wherein the domain handler outputs a semantic state, the method further comprising displaying context-relevant information suggesting at least one word to speak, the suggestion depending on the semantic state.
4. The method of claim 1 , further comprising:
storing the fragment for a delay period, using the fragment integrator, after detecting presence, wherein invoking the domain handler occurs after the delay period.
5. The method of claim 1 , wherein the domain handler causes a user interface to change on a display in real time.
6. The method of claim 1 , further comprising updating, using the domain handler, a portion of a semantic state.
7. A computer-implemented method comprising:
executing a fragment identifier on a portion of speech audio, the fragment identifier inferring presence of a fragment in the portion of speech audio and outputting the fragment obtained from its mapping of fragments to the fragment; and
executing a fragment integrator that receives the fragment from the fragment identifier, and matches the fragment against predefined rules:
responsive to the execution of the fragment identifier and the fragment integrator, invoking a domain handler using the fragment.
8. The method of claim 7 , further comprising updating a conversation state data structure with entity values from the fragment.
9. The method of claim 7 , wherein the domain handler outputs a semantic state, the method, further comprising displaying context-relevant information suggesting at least one word to speak, the suggestion depending on the semantic state.
10. The method of claim 7 , further comprising:
storing the fragment for a delay period, using the fragment integrator, after detecting the presence, wherein invoking the domain handler occurs after the delay period.
11. The method of claim 7 , wherein the domain handler causes a user interface to change on a display in real time.
12. The method of claim 7 , further comprising updating, using the domain handler, a portion of a semantic state.
13. The method of claim 12 , wherein the semantic state causes a context-dependent bias of the fragment identifier.
14. A computer-implemented method comprising:
transcribing audio, using a speech engine, to create a transcription;
executing a fragment identifier on a portion of the transcription, the fragment identifier detecting presence of a first fragment and outputting the first fragment; and
responsive to detecting the presence, invoking a domain handler with one fragment selected from the group consisting of the first fragment and a second fragment;
wherein the domain handler's updated semantic state influences subsequent transcriptions by the speech engine.
15. The method of claim 14 , further comprising updating a conversation state data structure with entity values from the fragment.
16. The method of claim 14 wherein the domain handler outputs a semantic state, the method, further comprising displaying context-relevant information suggesting at least one word to speak, the suggestion depending on the semantic state.
17. The method of claim 14 , further comprising:
storing the fragment for a delay period, using a fragment integrator, after detecting the presence, wherein invoking the domain handler occurs after the delay period.
18. The method of claim 14 , wherein the domain handler causes a user interface to change on a display in real time.
19. The method of claim 14 , further comprising updating, using the domain handler, a portion of a semantic state.
20. The method of claim 19 , wherein the semantic state causes a context-dependent bias of subsequent transcription by the speech engine.
21. A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of:
transcribing audio, using a speech engine, to create a transcription;
executing a fragment identifier on a portion of the transcription, the fragment identifier detecting a fragment and outputting the fragment; and
responsive to detecting, invoking a domain handler with the fragment.
22. The non-transitory computer-readable storage medium of claim 21 , the steps further comprising updating a conversation state data structure with entity values from the fragment.
23. The non-transitory computer-readable storage medium of claim 21 , wherein the domain handler outputs a semantic state, the steps further comprising displaying context-relevant information suggesting at least one word to speak, the suggestion depending on the semantic state.
24. The non-transitory computer-readable storage medium of claim 21 , the steps further comprising:
storing the fragment for a delay period, using a fragment integrator, after detecting presence, wherein invoking the domain handler occurs after the delay period.
25. The non-transitory computer-readable storage medium of claim 21 , wherein the domain handler causes a user interface to change on a display in real time.
26. The non-transitory computer-readable storage medium of claim 22 , the steps further comprising updating, using the domain handler, a portion of a semantic state.
27. The non-transitory computer-readable storage medium of claim 26 , wherein the semantic state causes a context-dependent bias of subsequent transcription by the speech engine.
28. A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of:
executing a fragment identifier on a portion of speech audio, the fragment identifier inferring presence of a fragment in the portion of speech audio and outputting the fragment obtained from its mapping of fragments to the fragment; and
responsive to the inferring, invoking a domain handler using the fragment.
29. The non-transitory computer-readable storage medium of claim 28 , the steps further comprising updating a conversation state data structure with entity values from the fragment.
30. The non-transitory computer-readable storage medium of claim 28 , wherein the domain handler outputs a semantic state, the steps further comprising displaying context-relevant information suggesting at least one word to speak, the suggestion depending on the semantic state.
31. The non-transitory computer-readable storage medium of claim 28 , the steps further comprising:
storing the fragment for a delay period, using a fragment integrator, after detecting the presence, wherein invoking the domain handler occurs after the delay period.
32. The non-transitory computer-readable storage medium of claim 28 , wherein the domain handler causes a user interface to change on a display in real time.
33. The non-transitory computer-readable storage medium of claim 29 , the steps further comprising updating, using the domain handler, a portion of a semantic state.
34. The non-transitory computer-readable storage medium of claim 33 , wherein the semantic state causes a context-dependent bias of the fragment identifier.