IP Library › Granted Patent US 12,178,641
Granted Patent B2
US 12,178,641 · App. 17/814,223 · Granted Dec 31, 2024

Systems and methods for fetus monitoring

Inventors: Shanhui Sun (Cambridge, MA); Ziyan Wu (Cambridge, MA); Xiao Chen (Cambridge, MA); Zhang Chen (Cambridge, MA); Yikang Liu (Cambridge, MA); Arun Innanje (Cambridge, MA); Terrence Chen (Cambridge, MA)
Assignee: SHANGHAI UNITED IMAGING INTELLIGENCE CO., LTD.
A61B8/0866A61B8/466A61B8/483A61B8/5207G06F3/011
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,178,641
App. No.
17/814,223
Granted
Dec 31, 2024
Kind
B2
Abstract

The present disclosure provides a system and method for fetus monitoring. The method may include obtaining ultrasound data relating to a fetus collected by an ultrasound imaging device; generating a 4D image of the fetus based on the ultrasound data; directing a display component of a virtual reality (VR) device to display the 4D image to an operator; detecting motion of the fetus based on the ultrasound data; and directing a haptic component of the VR device to provide haptic feedback with respect to the motion to the operator.

Claims (89)

1. A system, comprising:

at least one storage device storing a set of instructions for fetus monitoring; and

at least one processor configured to communicate with the at least one storage device, wherein when executing the set of instructions, the at least one processor is configured to direct the system to perform operations including:

obtaining ultrasound data relating to a fetus collected by an ultrasound imaging device;

generating at least one 4D image of the fetus based on the ultrasound data;

directing a display component of a virtual reality (VR) device to display the at least one 4D image to an operator;

detecting motion of at least part of the fetus based on the ultrasound data; and

directing a haptic component of the VR device to provide a feedback force with respect to the motion to the operator, wherein the at least one 4D image includes a first 3D fetus image and a second 3D fetus image captured prior to the first 3D fetus image, and the detecting the motion of the at least part of the fetus based on the ultrasound data includes:

determining a vertex correspondence between a plurality of first vertexes of a first mesh surface and a plurality of second vertexes of a second mesh surface, the first mesh surface representing the first 3D fetus image, and the second mesh surface representing the second 3D fetus image; and

for each of the plurality of first vertexes, determining motion information from its corresponding second vertex to the first vertex based on the vertex correspondence, wherein the at least one processor is configured to direct the system to determine a magnitude of the feedback force by:

determining a target area of the at least one 4D image;

determining, among the plurality of first vertexes, a plurality of target first vertexes in a portion of the first mesh surface that corresponds to the target area;

for each of the plurality of target first vertexes, determining a magnitude of a vertex force based on motion information of the target first vertex, the motion information of the target first vertex relating to the motion of the at least part of the fetus; and

determining the magnitude of the feedback force based on the magnitudes of the vertex forces of the plurality of target first vertexes.

2. The system of claim 1 , wherein the generating at least one 4D image of the fetus based on the ultrasound data includes:

generating multiple initial 3D images based on the ultrasound data relating to the fetus;

generating multiple 3D fetus images by segmenting a portion representing the fetus from each initial 3D image; and

generating the at least one 4D image based on the multiple 3D fetus images.

3. The system of claim 2 , wherein the generating the at least one 4D image based on the multiple 3D fetus images includes:

extracting a mesh surface from each 3D fetus image; and

rendering at least one 4D mesh surface including the multiple mesh surfaces to generate the at least one 4D image.

4. The system of claim 1 , wherein the determining

the vertex correspondence between the plurality of first vertexes of

the first mesh surface and the plurality of second vertexes of the second mesh surface includes:

determining a motion field between the first 3D fetus image and the second 3D fetus image; and

determining the vertex correspondence based on the motion field.

5. The system of claim 4 , wherein the motion field is determined based on an optical flow-based technique or a motion field determination model.

6. The system of claim 1 , wherein the determining

the vertex correspondence between the plurality of first vertexes of

the first mesh surface and the plurality of second vertexes of the second mesh surface includes:

generating a first point cloud representing the first 3D fetus image and a second point cloud representing the second 3D fetus image, wherein the first point cloud includes a plurality of first points, and the second point cloud includes a plurality of second points; and

determining the vertex correspondence by registering the first point cloud to the second point cloud.

7. The system of claim 1 , wherein the at least one processor is further configured to direct the system to perform operations including:

directing a speaker to play a sound relating to the fetus.

8. The system of claim 1 , wherein the ultrasound imaging device is a 4D ultrasound imaging device.

9. The system of claim 1 , wherein the motion information of the target first vertex includes a displacement of the target first vertex, and the determining the magnitude of the vertex force based on the motion information of the target first vertex includes:

determining the magnitude of the vertex force based on the displacement of the target first vertex.

10. The system of claim 1 , wherein, for each of the plurality of target first vertexes, the determining a magnitude of the vertex force based on the motion information of the target first vertex includes:

determining the magnitude of the vertex force based on Newton's second law according to Equation as follows:

F=ma,

wherein F denotes the vertex force, m denotes a vertex mass of the target first vertex, and a denotes an acceleration of the target first vertex.

11. The system of claim 10 , wherein the vertex mass is determined based on a size or a volume of the fetus.

12. The system of claim 1 , wherein the display component of the VR device includes a first display component corresponding to a left eye of the operator and a second display component corresponding to a right eye of the operator, and the at least one processor is configured to direct the system to perform operations including:

rendering a first image corresponding to a first eye view and a second image corresponding to a second eye view based on the at least one 4D image; and

directing the first display component to display the first image and the second display component to display the second image to the operator.

13. A method for fetus monitoring, implemented on a computing device having at least one processor and at least one storage device, the method comprising:

obtaining ultrasound data relating to a fetus collected by an ultrasound imaging device;

generating at least one 4D image of the fetus based on the ultrasound data;

directing a display component of a virtual reality (VR) device to display the at least one 4D image to an operator;

detecting motion of at least part of the fetus based on the ultrasound data; and

directing a haptic component of the VR device to provide a feedback force with respect to the motion to the operator, wherein the at least one 4D image includes a first 3D fetus image and a second 3D fetus image captured prior to the first 3D fetus image, and the detecting the motion of the at least part of the fetus based on the ultrasound data includes:

determining a vertex correspondence between a plurality of first vertexes of a first mesh surface and a plurality of second vertexes of a second mesh surface, the first mesh surface representing the first 3D fetus image, and the second mesh surface representing the second 3D fetus image; and

for each of the plurality of first vertexes, determining motion information from its corresponding second vertex to the first vertex based on the vertex correspondence, wherein the at least one processor is configured to direct the system to determine a magnitude of the feedback force by:

determining a target area of the at least one 4D image;

determining, among the plurality of first vertexes, a plurality of target first vertexes in a portion of the first mesh surface that corresponds to the target area;

for each of the plurality of target first vertexes, determining a magnitude of a vertex force based on motion information of the target first vertex, the motion information of the target first vertex relating to the motion of the at least part of the fetus; and

determining the magnitude of the feedback force based on the magnitudes of the vertex forces of the plurality of target first vertexes.

14. The method of claim 13 , wherein the generating at least one 4D image of the fetus based on the ultrasound data includes:

generating multiple initial 3D images based on the ultrasound data relating to the fetus;

generating multiple 3D fetus images by segmenting a portion representing the fetus from each initial 3D image; and

generating the at least one 4D image based on the multiple 3D fetus images.

15. The method of claim 14 , wherein the generating the at least one 4D image based on the multiple 3D fetus images includes:

extracting a mesh surface from each 3D fetus image; and

rendering at least one 4D mesh surface including the multiple mesh surfaces to generate the at least one 4D image.

16. The method of claim 13 , wherein the determining

the vertex correspondence between the plurality of first vertexes of

the first mesh surface and the plurality of second vertexes of the second mesh surface includes:

determining a motion field between the first 3D fetus image and the second 3D fetus image; and

determining the vertex correspondence based on the motion field.

17. The method of claim 16 , wherein the motion field is determined based on an optical flow-based technique or a motion field determination model.

18. The method of claim 13 , wherein the determining

the vertex correspondence between the plurality of first vertexes of

the first mesh surface and the plurality of second vertexes of the second mesh surface includes:

generating a first point cloud representing the first 3D fetus image and a second point cloud-representing the second 3D fetus image, wherein the first point cloud includes a plurality of first points, and the second point cloud includes a plurality of second points; and

determining the vertex correspondence by registering the first point cloud to the second point cloud.

19. The method of claim 13 , wherein the at least one processor is further configured to direct the system to perform operations including:

directing a speaker to play a sound relating to the fetus.

20. A non-transitory computer readable medium, comprising at least one set of instructions for fetus monitoring, wherein when executed by at least one processor of a computing device, the at least one set of instructions direct the at least one processor to perform operations including:

obtaining ultrasound data relating to a fetus collected by an ultrasound imaging device;

generating at least one 4D image of the fetus based on the ultrasound data;

directing a display component of a virtual reality (VR) device to display the at least one 4D image to an operator;

detecting motion of at least part of the fetus based on the ultrasound data; and

directing a haptic component of the VR device to provide a feedback force with respect to the motion to the operator, wherein the at least one 4D image includes a first 3D fetus image and a second 3D fetus image captured prior to the first 3D fetus image, and the detecting the motion of the at least prat of the fetus based on the ultrasound data includes:

determining a vertex correspondence between a plurality of first vertexes of a first mesh surface and a plurality of second vertexes of a second mesh surface, the first mesh surface representing the first 3D fetus image, and the second mesh surface representing the second 3D fetus image; and

for each of the plurality of first vertexes, determining motion information from its corresponding second vertex to the first vertex based on the vertex correspondence, wherein the at least one processor is configured to direct the system to determine a magnitude of the feedback force by:

determining a target area of the at least one 4D image;

determining, among the plurality of first vertexes, a plurality of target first vertexes in a portion of the first mesh surface that corresponds to the target area;

for each of the plurality of target first vertexes, determining a magnitude of a vertex force based on motion information of the target first vertex, the motion information of the target first vertex relating to the motion of the at least part of the fetus; and

determining the magnitude of the feedback force based on the magnitudes of the vertex forces of the plurality of target first vertexes.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 28, 2022
From: SUN, SHANHUI; WU, ZIYAN; CHEN, XIAO; CHEN, ZHANG; LIU, YIKANG; INNANJE, ARUN; CHEN, TERRENCE
To: UII AMERICA, INC.
Reel/Frame 061575/0273 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 28, 2022
From: UII AMERICA, INC.
To: SHANGHAI UNITED IMAGING INTELLIGENCE CO., LTD.
Reel/Frame 061575/0284 →
Continuity (1)
Related Publication 20240023925A1 · Jan 25, 2024
Cited By (1)
US 12,361,839