IP Library Granted Patent US 11,776,687
Granted Patent B2
US 11,776,687 · App. 17/094,564 · Granted Oct 3, 2023

Medical examination of human body using haptics

Inventors: Ramu Ramachandran (Bangalore, IN); Sankar Shanmugam (Bangalore, IN); Abilash Rajarethinam (Bangalore, IN)
Assignee: SONY GROUP CORPORATION
G16H40/67A61B5/0064A61B5/7267A61B5/7445A61B5/7455G06F3/016G06T15/08G16H30/20G16H30/40G16H50/20G16H50/50A61B5/0002A61B5/445A61B5/4504G06T2210/41
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,776,687
App. No.
17/094,564
Granted
Oct 3, 2023
Kind
B2
Abstract

An electronic apparatus and method for medical examination of human body using haptics is provided. The electronic apparatus controls a first head-mounted display to render a 3D model of an anatomical portion of the body of a human subject. The rendered 3D model includes a region corresponding to defect portion in the anatomical portion. The electronic apparatus transmits a touch input to wearable sensor in contact with the anatomical portion. Such an input corresponds to a human touch on the region of the rendered 3D model. The electronic apparatus receives, based on the touch input, bio-signals associated with the defect portion via the wearable sensor. The bio-signals include physiological signals and somatic sensation information associated with the defect portion. As a response to the human touch, the electronic apparatus controls a wearable haptic device to generate a haptic feedback based on the received set of bio-signals.

Claims (79)

1. An electronic apparatus, comprising:

circuitry communicatively coupled to a first head-mounted display and a wearable haptic device in contact with a user, wherein the circuitry is configured to:

control the first head-mounted display to render a first three-dimensional (3D) model of at least a first anatomical portion of a body of a human subject, wherein

the human subject is different from the user, and

the rendered first 3D model comprises a first region corresponding to a defect portion in the first anatomical portion of the body;

transmit a touch input to a wearable sensor in contact with at least the first anatomical portion of the body of the human subject, wherein

the transmitted touch input corresponds to a human touch, by the user, on the first region of the rendered first 3D model, and

the transmitted touch input includes at least a pressure signal indicating an amount of pressure applied during the human touch by the user;

receive, based on the touch input, a set of bio-signals associated with the defect portion via the wearable sensor, wherein the set of bio-signals comprises physiological signals and somatic sensation information associated with the defect portion;

control the wearable haptic device to generate a haptic feedback based on the received set of bio-signals, wherein the haptic feedback is generated as a response to the human touch;

receive a first set of features comprising anthropometric features related to the body of the human subject and medical condition information associated with the human subject;

generate an input feature for a machine learning model based on the received set of bio-signals, the anthropometric features related to the body of the human subject, and the medical condition information associated with the human subject; and

input the generated input feature to the machine learning model, wherein the machine learning model outputs a recommendation based on the input feature.

2. The electronic apparatus according to claim 1 , wherein the first region of the first 3D model corresponds to the defect portion, which is one of a tissue deformation or defect, a bruised tissue or skin, a fractured bone, a tumorous tissue or organ, a deformed bone, or a swelling or lump.

3. The electronic apparatus according to claim 1 , wherein the circuitry is further configured to:

receive depth information associated with the defect portion from one or more imaging sensors; and

generate the first region corresponding to the defect portion in the first anatomical portion of the body, based the received depth information.

4. The electronic apparatus according to claim 1 , wherein the circuitry is further configured to:

control a camera rig comprising one or more imaging sensors to scan at least the first anatomical portion of the body of the human subject;

receive 3D scan data of at least the first anatomical portion of the body of the human subject based on the scan; and

control the first head-mounted display to render the first 3D model of at least the first anatomical portion of the body based on the received 3D scan data.

5. The electronic apparatus according to claim 4 , wherein the 3D scan data comprises a 3D representation of at least one internal organ of the human subject.

6. The electronic apparatus according to claim 1 , wherein

the circuitry is further configured to generate the first 3D model based on the received first set of features, and

the generated first 3D model is rendered on the first head-mounted display.

7. The electronic apparatus according to claim 6 , wherein

the circuitry is further configured to update the generated first 3D model by a mesh deformation of at least the first region of the generated first 3D model,

the mesh deformation is applied based on the received second set of features, and

the updated first 3D model is rendered on the first head-mounted display.

8. The electronic apparatus according to claim 1 , wherein

the circuitry is further configured to receive a second set of features related to the defect portion in the first anatomical portion of the body of the human subject, and

the received second set of features comprises appearance attributes associated with the defect portion.

9. The electronic apparatus according to claim 8 , wherein the second set of features comprises at least one of physical deformities associated with at least the first anatomical portion of the body or deformities associated with one or more internal organs of the body.

10. The electronic apparatus according to claim 1 , wherein the somatic sensation information comprises touch sensation information associated with the defect portion, kinesthetic information associated with the defect portion, and haptic perception information associated with the defect portion.

11. The electronic apparatus according to claim 1 , wherein the recommendation comprises at least one of a course of treatment for the defect portion, a current condition of the defect portion, a prognosis for the defect portion, and a diagnosis related to the defect portion.

12. The electronic apparatus according to claim 1 , wherein the circuitry is further configured to:

extract the recommendation from the machine learning model; and

control the first head-mounted display to render the recommendation.

13. The electronic apparatus according to claim 1 , wherein

the circuitry is further configured to transmit the touch input to a haptic device in contact with at least the first anatomical portion of the body of the human subject, and

the haptic device:

receives the transmitted touch input; and

generates a haptic sensation on the first anatomical portion of the body, based on the received touch input.

14. The electronic apparatus according to claim 1 , wherein

the circuitry is further configured to control a second head-mounted display, as worn on a head of the human subject, to render the first 3D model and a second 3D model comprising a human hand,

the second 3D model is rendered to emulate a movement of the human hand while the human hand applies the human touch on the first region of the rendered 3D first model, and

the second head-mounted display is controlled based on the received touch input.

15. A method, comprising:

controlling a first head-mounted display to render a first three-dimensional (3D) model of at least a first anatomical portion of a body of a human subject,

wherein the rendered first 3D model comprises a first region corresponding to a defect portion in the first anatomical portion of the body;

transmitting a touch input to a wearable sensor in contact with at least the first anatomical portion of the body of the human subject, wherein

the transmitted touch input corresponds to a human touch, by a user, on the first region of the rendered first 3D model,

the human subject is different from the user, and

the transmitted touch input includes at least a pressure signal indicating an amount of pressure applied during the human touch by the user;

receiving, based on the touch input, a set of bio-signals associated with the defect portion via the wearable sensor, wherein the set of bio-signals comprises physiological signals and somatic sensation information associated with the defect portion;

controlling a wearable haptic device, in contact with the user, to generate a haptic feedback based on the received set of bio-signals, wherein the haptic feedback is generated as a response to the human touch;

receiving a first set of features comprising anthropometric features related to the body of the human subject and medical condition information associated with the human subject;

generating an input feature for a machine learning model based on the received set of bio-signals, the anthropometric features related to the body of the human subject, and the medical condition information associated with the human subject; and

inputting the generated input feature to the machine learning model, wherein the machine learning model outputs a recommendation based on the input feature.

16. The method according to claim 15 , further comprising:

extracting the recommendation from the machine learning model; and

controlling the first head-mounted display to render the recommendation.

17. The method according to claim 15 , further comprising:

transmitting the touch input to a haptic device in contact with at least the first anatomical portion of the body of the human subject;

receiving, by the haptic device, the transmitted touch input; and

generating, by the haptic device, a haptic sensation on the first anatomical portion of the body, based on the received touch input.

18. A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by an electronic apparatus, causes the electronic apparatus to execute operations, the operations comprising:

controlling a first head-mounted display to render a first three-dimensional (3D) model of at least a first anatomical portion of a body of a human subject,

wherein the rendered first 3D model comprises a first region corresponding to a defect portion in the first anatomical portion of the body;

transmitting a touch input to a wearable sensor in contact with at least the first anatomical portion of the body of the human subject, wherein

the transmitted touch input corresponds to a human touch, by a user, on the first region of the rendered first 3D model,

the human subject is different from the user, and

the transmitted touch input includes at least a pressure signal indicating an amount of pressure applied during the human touch by the user;

receiving, based on the touch input, a set of bio-signals associated with the defect portion via the wearable sensor, wherein the set of bio-signals comprises physiological signals and somatic sensation information associated with the defect portion;

controlling a wearable haptic device, in contact with the user, to generate a haptic feedback based on the received set of bio-signals, wherein the haptic feedback is generated as a response to the human touch;

receiving a first set of features comprising anthropometric features related to the body of the human subject and medical condition information associated with the human subject;

generating an input feature for a machine learning model based on the received set of bio-signals, the anthropometric features related to the body of the human subject, and the medical condition information associated with the human subject; and

inputting the generated input feature to the machine learning model, wherein the machine learning model outputs a recommendation based on the input feature.

19. The electronic apparatus according to claim 1 , wherein the wearable sensor measures at least one parameter associated with the human subject, based on the touch input transmitted to the wearable sensor.

Assignments (2)
CHANGE OF NAME Recorded Sep 10, 2021
From: SONY CORPORATION
To: SONY GROUP CORPORATION
Reel/Frame 057579/0449 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 10, 2020
From: RAMACHANDRAN, RAMU; SHANMUGAM, SANKAR; RAJARETHINAM, ABILASH
To: SONY CORPORATION
Reel/Frame 054328/0243 →
Continuity (1)
Related Publication 20220148723A1 · May 12, 2022