IP Library Granted Patent US 12,531,163
Granted Patent B2
US 12,531,163 · App. 17/666,817 · Granted Jan 20, 2026

Medical intelligence system and method

Inventors: Jeremy Martin Jancsary (Vienna, AT); Joel Praveen Pinto (Aachen, DE); Uwe Helmut Jost (Groton, MA); William F. Ganong, III (Brookline, MA)
Assignee: Microsoft Technology Licensing, LLC.
G16H80/00G16H10/60G16H20/10G16H30/40G16H40/20G16H40/63G16H40/67G16H50/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,531,163
App. No.
17/666,817
Granted
Jan 20, 2026
Kind
B2
Abstract

A method, computer program product, and computing system for: monitoring a meeting between a patient and a medical entity during a medical encounter; gathering information during the medical encounter, thus generating gathered encounter information, wherein the gathered encounter information includes audio-based content of the patient; generating audio-based content information via artificial intelligence, wherein the audio-based content information is based at least in part upon the audio-based content and/or the gathered encounter information and is configured to provide guidance to the medical entity concerning the audio-based content; and providing the audio-based content information to the medical entity.

Claims (55)

1 . A computer-implemented method, executed on a computing system, comprising:

monitoring, by an ambient cooperative intelligence (ACI) system, a medical encounter between a patient and a medical entity in a clinical environment;

detecting, by one or more machine vision devices of the ACI system, a position of the patient within the clinical environment, the ACI system including discrete audio acquisition devices forming a microphone array within the clinical environment;

steering, by the ACI system, one or more audio recording beams of the one or more discrete audio acquisition devices toward the detected position of the patient within the clinical environment during the medical encounter, wherein the one or more discrete audio acquisition devices are configured to directionally record audio-based content of the patient using the one or more audio recording beams;

processing, via artificial intelligence of the ACI system without human intervention, the audio-based content to identify one or more potential conditions of the patient, wherein processing the audio-based content includes comparing the audio-based content to information associated with one or more ailments to identify the one or more potential conditions of the patient; and

providing, by the ACI system, an indication of the one or more potential conditions of the patient to the medical entity, the indication of the one or more potential conditions of the patient being provided to guide the medical entity in diagnosing the patient.

2 . The computer-implemented method of claim 1 , wherein the medical entity is a medical professional.

3 . The computer-implemented method of claim 1 , wherein the medical encounter includes one or more of:

an intake portion;

a consultation portion; and

a follow-up portion.

4 . The computer-implemented method of claim 1 , wherein the indication of the one or more potential conditions of the patient is displayed to the medical entity via a transparent overlay informational window, the transparent overlay informational window superimposed over other images or information such that the transparent overlay informational window does not obscure the other images or information.

5 . The computer-implemented method of claim 4 , wherein the transparent overlay informational window is configured as a heads up display.

6 . A computer program product residing on a computer readable medium having programming instructions stored thereon which, when executed by one or more processors of an ambient cooperative intelligence (ACI) system, cause the ACI system to perform operations comprising:

monitoring a medical encounter between a patient and a medical entity in a clinical environment;

detecting, by one or more machine vision devices of the ACI system, a position of the patient within the clinical environment, the ACI system including discrete audio acquisition devices forming a microphone array within the clinical environment;

steering one or more audio recording beams of the one or more discrete audio acquisition devices toward the detected position of the patient within the clinical environment during the medical encounter, wherein the one or more discrete audio acquisition devices are configured to directionally record audio-based content of the patient using the one or more audio recording beams;

processing, via artificial intelligence of the ACI system without human intervention, the audio-based content to identify one or more potential conditions of the patient, wherein processing the audio-based content includes comparing the audio-based content to information associated with one or more ailments to identify the one or more potential conditions of the patient; and

providing an indication of the one or more potential conditions of the patient to the medical entity, the indication of the one or more potential conditions of the patient being provided to guide the medical entity in diagnosing the patient.

7 . The computer program product of claim 6 , wherein the medical entity is a medical professional.

8 . The computer program product of claim 6 , wherein the medical encounter includes one or more of:

an intake portion;

a consultation portion; and

a follow-up portion.

9 . The computer program product of claim 6 , wherein the indication of the one or more potential conditions of the patient is displayed to the medical entity via a transparent overlay informational window, the transparent overlay informational window superimposed over other images or information such that the transparent overlay informational window does not obscure the other images or information.

10 . The computer program product of claim 9 , wherein the transparent overlay informational window is configured as a heads up display.

11 . The computer program product of claim 6 , wherein generating audio-based content information via artificial intelligence includes:

comparing a current version of the audio-based content to a previous version of the audio-based content to determine if the audio-based content has changed.

12 . An ambient cooperative intelligence (ACI) system including one or more processors and a computer readable medium having programming instructions stored thereon which, when executed by one or more processors, cause the ACI system to perform operations comprising:

monitoring a medical encounter between a patient and a medical entity in a clinical environment;

detecting, by one or more machine vision devices of the ACI system, a position of the patient within the clinical environment, the ACI system including discrete audio acquisition devices forming a microphone array within the clinical environment;

steering one or more audio recording beams of the one or more discrete audio acquisition devices toward the detected position of the patient within the clinical environment during the medical encounter, wherein the one or more discrete audio acquisition devices are configured to directionally record audio-based content of the patient using the one or more audio recording beams;

processing, via artificial intelligence of the ACI system without human intervention, the audio-based content to identify one or more potential conditions of the patient, wherein processing the audio-based content includes comparing the audio-based content to information associated with one or more ailments to identify the one or more potential conditions of the patient; and

providing an indication of the one or more potential conditions of the patient to the medical entity, the indication of the one or more potential conditions of the patient being provided to guide the medical entity in diagnosing the patient.

13 . The ACI system of claim 12 , wherein the medical entity is a medical professional.

14 . The ACI system of claim 12 , wherein the medical encounter includes one or more of:

an intake portion;

a consultation portion; and

a follow-up portion.

15 . The ACI system of claim 12 , wherein the indication of the one or more potential conditions of the patient is displayed to the medical entity via a transparent overlay informational window, the transparent overlay informational window superimposed over other images or information such that the transparent overlay informational window does not obscure the other images or information.

16 . The ACI system of claim 15 , wherein the transparent overlay informational window is configured as a heads up display.

17 . The ACI system of claim 12 , wherein generating audio-based content information via artificial intelligence includes:

comparing a current version of the audio-based content to a previous version of the audio-based content to determine if the audio-based content has changed.

18 . The ACI system of claim 12 , wherein the one or more machine vision devices of the ACI system include red, green, blue (RGB) sensors.

19 . The ACI system of claim 12 , wherein the one or more machine vision devices of the ACI system include infrared sensors.

20 . The ACI system of claim 12 , wherein the one or more machine vision devices of the ACI system include ultraviolet sensors.

21 . The ACI system of claim 12 , wherein the one or more machine vision devices of the ACI system include SONAR sensors, RADAR sensors, or thermal imaging sensors.

22 . The computer program product of claim 6 , wherein the one or more machine vision devices of the ACI system include red, green, blue (RGB) sensors.

23 . The computer program product of claim 6 , wherein the one or more machine vision devices of the ACI system include infrared sensors.

24 . The computer program product of claim 6 , wherein the one or more machine vision devices of the ACI system include ultraviolet sensors.

25 . The computer program product of claim 6 , wherein the one or more machine vision devices of the ACI system include SONAR sensors, RADAR sensors, or thermal imaging sensors.

26 . The computer-implemented method of claim 1 , wherein the one or more machine vision devices of the ACI system include red, green, blue (RGB) sensors.

27 . The computer-implemented method of claim 1 , wherein the one or more machine vision devices of the ACI system include infrared sensors.

28 . The computer-implemented method of claim 1 , wherein the one or more machine vision devices of the ACI system include ultraviolet sensors.

29 . The computer-implemented method of claim 1 , wherein the one or more machine vision devices of the ACI system include SONAR sensors, RADAR sensors, or thermal imaging sensors.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 14, 2023
From: NUANCE COMMUNICATIONS, INC.
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 065578/0676 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 26, 2022
From: JANCSARY, JEREMY MARTIN; PINTO, JOEL PRAVEEN; JOST, UWE HELMUT; GANONG, WILLIAM F., III
To: NUANCE COMMUNICATIONS, INC.
Reel/Frame 060913/0986 →
Continuity (2)
Provisional Application 63148458 · Feb 11, 2021
Related Publication 20220254496A1 · Aug 11, 2022
References Cited (117)
US 8548828B1 · Longmire · 2013 [cited by applicant]
US 10628553B1 · Murrish · 2020 [cited by applicant]
US 11488712B2 · Lee et al. · 2022 [cited by applicant]
US 11515021B2 · Mason · 2022 [cited by applicant]
US 11742092B2 · Murrish et al. · 2023 [cited by applicant]
US 20020115912A1 · Muraki · 2002 [cited by applicant]
US 20030110059A1 · John, III · 2003 [cited by applicant]
US 20030181790A1 · David · 2003 [cited by applicant]
US 20050055246A1 · Simon · 2005 [cited by applicant]
US 20050149364A1 · Ombrellaro · 2005 [cited by applicant]
US 20050273363A1 · Lipscher et al. · 2005 [cited by applicant]
US 20060149597A1 · Powell · 2006 [cited by applicant]
US 20070127795A1 · Lau et al. · 2007 [cited by applicant]
US 20070234219A1 · Bhattaru · 2007 [cited by applicant]
US 20080242977A1 · Sirohey et al. · 2008 [cited by applicant]
US 20090138285A1 · Denberg · 2009 [cited by applicant]
US 20090259492A1 · Cossman · 2009 [cited by applicant]
US 20100109860A1 · Williamson et al. · 2010 [cited by applicant]
US 20110166465A1 · Clements et al. · 2011 [cited by applicant]
US 20110246225A1 · Green et al. · 2011 [cited by applicant]
US 20120173281A1 · Dilella et al. · 2012 [cited by applicant]
US 20150154355A1 · Lin · 2015 [cited by applicant]
US 20150213224A1 · Amarasingham et al. · 2015 [cited by applicant]
US 20160071547A1 · Sugawara · 2016 [cited by applicant]
US 20160110504A1 · Fialkov · 2016 [cited by applicant]
US 20160128647A1 · King · 2016 [cited by applicant]
US 20160140318A1 · Stangel · 2016 [cited by applicant]
US 20160148375A1 · Oh et al. · 2016 [cited by applicant]
US 20160180743A1 · Ahmad · 2016 [cited by applicant]
US 20160188843A1 · Staples, II et al. · 2016 [cited by applicant]
US 20160302666A1 · Shaya · 2016 [cited by applicant]
US 20170116384A1 · Ghani · 2017 [cited by applicant]
US 20170235905A1 · Santiago, Jr. · 2017 [cited by applicant]
US 20170329922A1 · Eberting · 2017 [cited by applicant]
US 20180121606A1 · Allen · 2018 [cited by applicant]
US 20180121728A1 · Wells · 2018 [cited by applicant]
US 20180181712A1 · Ensey et al. · 2018 [cited by applicant]
US 20180211730A1 · Slepian et al. · 2018 [cited by applicant]
US 20180226158A1 · Fish et al. · 2018 [cited by applicant]
US 20180247024A1 · Divine et al. · 2018 [cited by applicant]
US 20180261307A1 · Couse et al. · 2018 [cited by applicant]
US 20180316947A1 · Todd · 2018 [cited by applicant]
US 20180330059A1 · Bates et al. · 2018 [cited by applicant]
US 20180356244A1 · Han et al. · 2018 [cited by applicant]
US 20190065970A1 · Bonutti et al. · 2019 [cited by applicant]
US 20190156921A1 · Kohli · 2019 [cited by applicant]
US 20190189259A1 · Clark · 2019 [cited by applicant]
US 20190272147A1 · Vozila et al. · 2019 [cited by applicant]
US 20190272901A1 · Almendro Barreda et al. · 2019 [cited by applicant]
US 20200004561A1 · Kottler et al. · 2020 [cited by applicant]
US 20200066414A1 · Neff et al. · 2020 [cited by applicant]
US 20200221951A1 · Amble · 2020 [cited by examiner]
US 20200228616A1 · Nishikawa · 2020 [cited by applicant]
US 20200258510A1 · Lavery et al. · 2020 [cited by applicant]
US 20200279659A1 · De Brouwer et al. · 2020 [cited by applicant]
US 20200398063A1 · Debates et al. · 2020 [cited by applicant]
US 20200402656A1 · Debates · 2020 [cited by applicant]
US 20210082554A1 · Kalia et al. · 2021 [cited by applicant]
US 20210118555A1 · Dreyer · 2021 [cited by applicant]
US 20210142882A1 · Zhang · 2021 [cited by applicant]
US 20210142903A1 · Mason · 2021 [cited by examiner]
US 20210182932A1 · Bello, Jr. · 2021 [cited by applicant]
US 20210295963A1 · Bakshi · 2021 [cited by applicant]
US 20210335503A1 · Williams, III · 2021 [cited by applicant]
US 20210345934A1 · Landgraf · 2021 [cited by examiner]
US 20210375489A1 · Tsushita · 2021 [cited by applicant]
US 20210383909A1 · Mason · 2021 [cited by examiner]
US 20220254480A1 · Jancsary et al. · 2022 [cited by applicant]
US 20220254485A1 · Jancsary et al. · 2022 [cited by applicant]
US 20220254495A1 · Jancsary et al. · 2022 [cited by applicant]
US 20220254514A1 · Jancsary et al. · 2022 [cited by applicant]
US 20220254515A1 · Jancsary et al. · 2022 [cited by applicant]
US 20220254516A1 · Jancsary et al. · 2022 [cited by applicant]
US 20220254517A1 · Jancsary et al. · 2022 [cited by applicant]
CN 110024026A · 2019 [cited by applicant]
EP 3719809A1 · 2020 [cited by applicant]
JP 2019101715A · 2019 [cited by applicant]
WO 2014205079A2 · 2014 [cited by applicant]
WO WO2017158338A1 · 2017 [cited by examiner]
“Non Final Office Action Issued in U.S. Appl. No. 17/666,780”, Mailed Date: Oct. 4, 2023, 23 Pages. [cited by applicant]
“Non Final Office Action Issued in U.S. Appl. No. 17/666,828”, Mailed Date: Sep. 13, 2023, 19 Pages. [cited by applicant]
“Non Final Office Action Issued in U.S. Appl. No. 17/666,847”, Mailed Date: Sep. 27, 2023, 41 Pages. [cited by applicant]
“Non Final Office Action Issued in U.S. Appl. No. 17/666,854”, Mailed Date: Sep. 13, 2023, 19 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/015642”, Mailed Date: Jun. 2, 2022, 9 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/015649”, Mailed Date: Apr. 19, 2022, 7 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/015651”, Mailed Date: Apr. 28, 2022, 7 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/015654”, Mailed Date: Jun. 1, 2022, 7 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/015657”, Mailed Date: May 3, 2022, 8 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/015660”, Mailed Date: May 23, 2022, 9 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/015667”, Mailed Date: May 24, 2022, 7 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/015671”, Mailed Date: May 23, 2022, 9 Pages. [cited by applicant]
Final Office Action mailed on Oct. 1, 2024, in U.S. Appl. No. 17/666,787, 17 pages. [cited by applicant]
Notice of Allowance mailed on Oct. 9, 2024, in U.S. Appl. No. 17/666,828, 9 pages. [cited by applicant]
Notice of Allowance mailed on Sep. 30, 2024, in U.S. Appl. No. 17/666,780, 10 pages. [cited by applicant]
Final Office Action mailed on Aug. 12, 2024, in U.S. Appl. No. 17/666,847, 58 pages. [cited by applicant]
Final Office Action mailed on Aug. 19, 2024, in U.S. Appl. No. 17/666,839, 17 pages. [cited by applicant]
Final Office Action mailed on Jul. 18, 2024, in U.S. Appl. No. 17/666,780, 28 pages. [cited by applicant]
Notice of Allowance mailed on Sep. 12, 2024, in U.S. Appl. No. 17/666,854, 10 pages. [cited by applicant]
Final Office Action mailed on Feb. 28, 2024, in U.S. Appl. No. 17/666,828, 18 pages. [cited by applicant]
Final Office Action mailed on Jan. 12, 2024, in U.S. Appl. No. 17/666,847, 46 pages. [cited by applicant]
Final Office Action mailed on Jan. 19, 2024, in U.S. Appl. No. 17/666,780, 24 pages. [cited by applicant]
Non-Final Office Action mailed on Dec. 20, 2023, in U.S. Appl. No. 17/666,839, 15 pages. [cited by applicant]
Final Office Action mailed on Mar. 27, 2024, in U.S. Appl. No. 17/666,854, 21 pages. [cited by applicant]
Non-Final Office Action issued in U.S. Appl. No. 17/666,828, mailed on Jun. 20, 2024, 26 Pages. [cited by applicant]
Non-Final Office Action mailed on Jun. 6, 2024, in U.S. Appl. No. 17/666,787, 13 pages. [cited by applicant]
Non-Final Office Action mailed on Mar. 22, 2024, in U.S. Appl. No. 17/666,847, 55 pages. [cited by applicant]
Non-Final Office Action mailed on Mar. 27, 2024, in U.S. Appl. No. 17/666,780, 27 pages. [cited by applicant]
Communication pursuant to Rules 70(2) and 70a(2) EPC Received for European Application No. 22753207.4, mailed on Dec. 10, 2024, 01 pages. [cited by applicant]
Extended European Search Report Received for European Application No. 22753207.4, mailed on Nov. 22, 2024, 10 pages. [cited by applicant]
U.S. Appl. No. 17/666,780, filed Feb. 8, 2022. [cited by applicant]
U.S. Appl. No. 17/666,828, filed Feb. 8, 2022. [cited by applicant]
U.S. Appl. No. 17/666,787, filed Feb. 8, 2022. [cited by applicant]
U.S. Appl. No. 17/666,798, filed Feb. 8, 2022. [cited by applicant]
U.S. Appl. No. 17/666,839, filed Feb. 8, 2022. [cited by applicant]
U.S. Appl. No. 17/666,847, filed Feb. 8, 2022. [cited by applicant]
U.S. Appl. No. 17/666,854, filed Feb. 8, 2022. [cited by applicant]
Non-Final Office Action mailed on Mar. 20, 2025, in U.S. Appl. No. 17/666,798, 16 pages. [cited by applicant]