IP Library › Granted Patent US 12,608,408
Granted Patent B2
US 12,608,408 · App. 19/178,757 · Granted Apr 21, 2026

Machine learning-based evaluation of recorded interactions

Inventors: Aashraya Sachdeva (Mohali, IN); Digvijay Anil Ingle (Akola, IN); Mayank Sati (Nainital, IN); Surya Prakash Sahu (Khorda, IN); Cijo George (Bengaluru, IN); Claudia Lee (El Dorado Hills, CA); Bhasker Sharma (Bengaluru, IN); Jithendra Vepa (Bengaluru, IN)
Assignee: Observe.AI, Inc.
G06F16/33295G06F16/3326
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,608,408
App. No.
19/178,757
Granted
Apr 21, 2026
Kind
B2
Abstract

Machine learning-based evaluation of recorded interactions is disclosed, including: obtaining an evaluation plan to correspond to a new question; retrieving a representative interaction based at least in part on the new question; using a reasoning and answer language model to evaluate the representative interaction against the new question based at least in part on the evaluation plan and to provide a preview evaluation result; outputting, at a user interface, the new question and the preview evaluation result of the representative interaction; receiving, via the user interface, user feedback to the preview evaluation result; updating the reasoning and answer language model based at least in part on the user feedback to the preview evaluation result; and storing a feedback data set including the user feedback.

Claims (72)

1 . A system, comprising:

one or more processors configured to:

obtain an evaluation plan to correspond to a new question, wherein to obtain the evaluation plan further comprises to:

analyze the new question using a large language model (LLM) to determine key characteristics;

compare the key characteristics against predefined criteria associated with a library of predetermined evaluation plans to determine a set of candidate evaluation plans;

determine a set of accuracies corresponding to respective ones of the set of candidate evaluation plans; and

select the evaluation plan from the set of candidate evaluation plans based at least in part on the set of accuracies;

retrieve a representative interaction based at least in part on the new question;

use a reasoning and answer language model to evaluate the representative interaction against the new question based at least in part on the evaluation plan and to provide a preview evaluation result;

output, at a user interface, the new question and the preview evaluation result of the representative interaction;

receive, via the user interface, user feedback to the preview evaluation result; and

update the reasoning and answer language model based at least in part on the user feedback to the preview evaluation result; and

a storage device configured to store a feedback data set including the user feedback.

2 . The system of claim 1 , wherein the one or more processors are further configured to:

receive, via the user interface, a user submitted question;

determine based at least in part on the key characteristics that additional information is needed to clarify the user submitted question; and

present a request for the additional information at the user interface.

3 . The system of claim 2 , wherein the one or more processors are further configured to:

receive, via the user interface, the additional information; and

synthesize the user submitted question and the additional information into a refined question, wherein the new question comprises the refined question.

4 . The system of claim 1 , wherein to obtain the evaluation plan further comprises to prompt the LLM to generate one or more candidate evaluation plans to include in the set of candidate evaluation plans based at least in part on the new question.

5 . The system of claim 4 , wherein the LLM comprises a first LLM, wherein the reasoning and answer language model comprises a second LLM, wherein the first LLM includes more parameters than the second LLM.

6 . The system of claim 1 , wherein the LLM comprises a first LLM, and wherein the user feedback comprises a first user feedback, wherein the one or more processors are further configured to:

prompt a second LLM to generate one or more candidate evaluation plans to include in the set of candidate evaluation plans;

present the one or more candidate evaluation plans at the user interface;

receive a second user feedback to the one or more candidate evaluation plans; and

update the second LLM.

7 . The system of claim 1 , wherein to retrieve the representative interaction based at least in part on the new question comprises to:

determine a question embedding corresponding to the new question using an embedding model;

determine interaction embeddings corresponding to historical interactions using the embedding model; and

select the representative interaction by comparing the question embedding to the interaction embeddings.

8 . The system of claim 1 , wherein the evaluation plan comprises a set of criteria to be met by a given interaction to result in a particular answer to the new question.

9 . The system of claim 1 , wherein the one or more processors are further configured to store a mapping between the new question and the evaluation plan.

10 . The system of claim 9 , wherein the one or more processors are further configured to:

receive a new interaction;

determine to evaluate the new interaction using the new question;

obtain the evaluation plan corresponding to the new question in the mapping; and

use the updated reasoning and answer language model to evaluate the new interaction against the new question based at least in part on the evaluation plan and to provide a predicted evaluation result.

11 . The system of claim 10 , wherein the user interface comprises a first user interface, wherein the user feedback comprises a first user feedback, and wherein the one or more processors are further configured to:

present the new question and the predicted evaluation result at a second user interface;

receive, via the second user interface, a second user feedback to the predicted evaluation result; and

update the updated reasoning and answer language model based at least in part on the second user feedback to the predicted evaluation result.

12 . The system of claim 11 , wherein the predicted evaluation result comprises an answer to the new question, a set of reasoning that supports the answer, and a set of evidences from the new interaction that supports the set of reasoning.

13 . The system of claim 12 , wherein the second user feedback to the predicted evaluation result comprises a correction to at least one of the answer, the set of reasoning, and the set of evidences.

14 . A method, comprising:

obtaining an evaluation plan to correspond to a new question, wherein obtaining the evaluation plan further comprises:

analyzing the new question using a large language model (LLM) to determine key characteristics;

comparing the key characteristics against predefined criteria associated with a library of predetermined evaluation plans to determine a set of candidate evaluation plans;

determining a set of accuracies corresponding to respective ones of the set of candidate evaluation plans; and

selecting the evaluation plan from the set of candidate evaluation plans based at least in part on the set of accuracies;

retrieving a representative interaction based at least in part on the new question;

using a reasoning and answer language model to evaluate the representative interaction against the new question based at least in part on the evaluation plan and to provide a preview evaluation result;

outputting, at a user interface, the new question and the preview evaluation result of the representative interaction;

receiving, via the user interface, user feedback to the preview evaluation result;

updating the reasoning and answer language model based at least in part on the user feedback to the preview evaluation result; and

storing a feedback data set including the user feedback.

15 . The method of claim 14 , wherein the LLM comprises a first LLM, and wherein the user feedback comprises a first user feedback, the method further comprising:

prompting a second LLM to generate one or more candidate evaluation plans to include in the set of candidate evaluation plans;

presenting the one or more candidate evaluation plans at the user interface;

receiving a second user feedback to the one or more candidate evaluation plans; and

updating the second LLM.

16 . The method of claim 14 , wherein retrieving the representative interaction based at least in part on the new question comprises:

determining a question embedding corresponding to the new question using an embedding model;

determining interaction embeddings corresponding to historical interactions using the embedding model; and

selecting the representative interaction by comparing the question embedding to the interaction embeddings.

17 . The method of claim 14 , wherein the evaluation plan comprises a set of criteria to be met by a given interaction to result in a particular answer to the new question.

18 . The method of claim 14 , further comprising storing a mapping between the new question and the evaluation plan.

19 . The method of claim 18 , further comprising:

receiving a new interaction;

determining to evaluate the new interaction using the new question;

obtaining the evaluation plan corresponding to the new question in the mapping; and

using the updated reasoning and answer language model to evaluate the new interaction against the new question based at least in part on the evaluation plan and to provide a predicted evaluation result.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 15, 2025
From: SACHDEVA, AASHRAYA; INGLE, DIGVIJAY ANIL; SATI, MAYANK; SAHU, SURYA PRAKASH; GEORGE, CIJO; LEE, CLAUDIA; SHARMA, BHASKER; VEPA, JITHENDRA
To: OBSERVE.AI, INC.
Reel/Frame 072032/0829 →
Continuity (2)
Provisional Application 63685522 · Aug 21, 2024
Related Publication 20260056986A1 · Feb 26, 2026
References Cited (38)
US 12204524B1 · Birru · 2025 [cited by examiner]
US 12326869B1 · Alperin · 2025 [cited by examiner]
US 20250124060A1 · Hackman · 2025 [cited by examiner]
US 20250245233A1 · Jain · 2025 [cited by examiner]
Abdin et al. “Phi-3 technical report: A highly capable language model locally on your phone.” arXiv preprint arXiv:2404.14219 (2024). [cited by applicant]
Achiam et al. “Gpt-4 technical report.” arXiv preprint arXiv:2303.08774 (2023). [cited by applicant]
Ali et al. “Word error rate estimation for speech recognition: e-WER.” Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (vol. 2: Short Papers). Association for Computational Lingui… [cited by applicant]
Anonymous, “Introducing the next generation of Claude”, Mar. 4, 2024 from https://www.anthropic.com/news/claude-3-family, access on Apr. 14, 2025. [cited by applicant]
Chen et al. “Learning to maximize mutual information for chain-of-thought distillation.” arXiv preprint arXiv:2403.03348 (2024). [cited by applicant]
Chen et al. “Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.” arXiv preprint arXiv:2211.12588 (2022). [cited by applicant]
Deng et al. “Implicit chain of thought reasoning via knowledge distillation.” arXiv preprint arXiv:2311.01460 (2023). [cited by applicant]
Fu et al. “Entity-level sentiment analysis in contact center telephone conversations.” arXiv preprint arXiv:2210.13401 (2022). [cited by applicant]
Gemini Team et al. “Gemini: a family of highly capable multimodal models.” arXiv preprint arXiv:2312.11805 (2023). [cited by applicant]
Gemma Team et al. “Gemma: Open models based on gemini research and technology.” arXiv preprint arXiv:2403.08295 (2024). [cited by applicant]
Girish et al. “Interpretabilty of Speech Emotion Recognition modelled using Self-Supervised Speech and Text Pre-Trained Embeddings.” Interspeech. 2022. [cited by applicant]
Guruju et al., “Addressing Compliance in Call Centers with Entity Extraction.” Proc. Interspeech 2021 (2021):2347-2348. [cited by applicant]
Ingle et al. “Listening to Silences In Contact Center Conversations Using Textual Cues.” Proc. Interspeech 2023. 2023. [cited by applicant]
Kim et al. “The CoT collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning.” arXiv preprint arXiv:2305.14045 (2023). [cited by applicant]
Li et al. “Symbolic chain-of-thought distillation: Small models can also “think” step-by-step.” arXiv preprint arXiv:2306.14050 (2023). [cited by applicant]
Luo et al. “BioGPT: generative pre-trained transformer for biomedical text generation and mining.” Briefings in bioinformatics 23.6 (2022): bbac409. [cited by applicant]
Nathan et al. “Can probing classifiers reveal the learning by contact center large language models?: No, it doesn't !.” Proceedings of the Fifth Workshop on Insights from Negative Results in NLP. 2024. [cited by applicant]
Petroni, et al. “Language models as knowledge bases?.” arXiv preprint arXiv:1909.01066 (2019). [cited by applicant]
Roberts et al. “How much knowledge can you pack into the parameters of a language model?.” arXiv preprint arXiv:2002.08910 (2020). [cited by applicant]
Roy et al., “QART: A System for Real-Time Holistic Quality Assurance for Contact Center Dialogues”, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16) 2016. [cited by applicant]
Roziere et al. “Code llama: Open foundation models for code.” arXiv preprint arXiv:2308.12950 (2023). [cited by applicant]
Singhal et al. “Large language models encode clinical knowledge.” Nature 620.7972 (2023): 172-180. [cited by applicant]
Touvron et al., “Llama: Open and efficient foundation language models.” arXiv preprint arXiv:2302.13971 (2023). [cited by applicant]
Van Veen et al. “Adapted large language models can outperform medical experts in clinical text summarization.” Nature medicine 30.4 (2024): 1134-1142. [cited by applicant]
Wang, et al. “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.” arXiv preprint arXiv:2109.00859 (2021). [cited by applicant]
Wei et al. “Chain-of-thought prompting elicits reasoning in large language models.” Advances in neural information processing systems 35 (2022): 24824-24837. [cited by applicant]
Wei et al. “Finetuned language models are zero-shot learners.” arXiv preprint arXiv:2109.01652 (2021). [cited by applicant]
Wolf et al. “Transformers: State-of-the-art natural language processing.” Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. 2020. [cited by applicant]
Wu et al. “BloombergGPT: A large language model for finance.” arXiv preprint arXiv:2303.17564 (2023). [cited by applicant]
Yao et al. “Tree of thoughts: Deliberate problem solving with large language models.” Advances in neural information processing systems 36 (2023): 11809-11822. [cited by applicant]
Yu et al. “Kola: Carefully benchmarking world knowledge of large language models.” arXiv preprint arXiv:2306.09296 (2023). [cited by applicant]
Yuan et al. “Large language models illuminate a progressive pathway to artificial intelligent healthcare assistant.” Medicine Plus (2024): 100030. [cited by applicant]
Zhang, et al. “Sentiment analysis in the era of large language models: A reality check.” arXiv preprintarXiv:2305.15005 (2023). [cited by applicant]
Zhu et al. “Multilingual machine translation with large language models: Empirical results and analysis.” arXiv preprint arXiv:2304.04675 (2023). [cited by applicant]