Granted Patent B1

US 12,380,732 · App. 19/092,775 · Granted Aug 5, 2025

Inventors: Xin Ma (Jinan, CN); Xiaochen Huang (Jinan, CN); Yibin Li (Jinan, CN)

Assignee: Shandong University

G06V40/176A61B5/4082G06T7/0014G06V10/761G06V10/82G06V20/46G06V20/49G06V40/171G06V40/172G06T2207/10016G06T2207/30201

Abstract

Claims (68)

1. An auxiliary diagnosis method for Parkinson's disease (PD) based on static and dynamic features of facial expressions, comprising:

acquiring video data of various facial expressions performed by a to-be-tested patient;

pre-processing the video data to extract a plurality of optimal facial expression images corresponding to the various facial expressions;

synthesizing, using a generative network, a happy facial expression image of the to-be-tested patient in a healthy state to obtain a synthesized happy facial expression image; performing, based on a neutral facial expression image, a similarity discrimination, on the synthesized happy facial expression image and an extracted happy facial expression image to obtain similarity features; calculating distances between multiple facial key points in the various facial expression images to obtain a plurality of key features; splicing the similarity features and the plurality of key features to form static features;

calculating, based on the plurality of optimal facial expression images, coordinate change degrees of multiple facial key points of eyelids and mouth to obtain dynamic features; and

equilibrating dimensions of the static features and the dynamic features, followed by feature splicing using a static-dynamic feature balanced classification network to obtain spliced features; and outputting a classification prediction result of PD based on the spliced features;

wherein the video data is pre-processed through steps of:

splitting the video into a plurality of independent videos, each of the plurality of independent videos corresponds to a facial expression;

annotating facial key points for consecutive video-frame images of each of the plurality of independent videos; convert coordinates of each facial key point to relative coordinates based on a relatively fixed point; and performing normalization processing on the video-frame images;

uniformly selecting K images from a neutral-expression independent video; and calculating an average value of coordinates of each facial key point in the K images to obtain key point coordinates of an average neutral face; and

for video-frame images in each expression-specific independent video, calculating a distance between each facial key point in each video-frame image and a key point in the average neutral face corresponding thereto; sorting video-frame images of each expression-specific independent video in a descending order according to a sum of distances of all key points in each video-frame image to screen the first L images as the optimal facial expression state images;

the various facial expressions comprise a neutral facial expression, a happy facial expression, a sad facial expression, a surprised facial expression, a fearful facial expression, an angry facial expression, and a disgusted facial expression; and

the facial key points comprise relatively fixed points and dynamic flexible points, the relatively fixed points comprise points around a nose;

the similarity discrimination is performed through steps of:

acquiring M synthetic happy facial expression images and N extracted happy facial expression images;

extracting coordinates of facial key points around a mouth from the acquired images; and

calculating Euclidean distance between the same facial key point in each synthetic image and each extracted image to obtain similarity discrimination results, and the discrimination results are configured as similarity features;

the plurality of key features are obtained through steps of:

for a happy facial expression image, calculating distance variations between points near mouth corners and the relatively fixed points, and distance changes between an upper lip and a lower lip;

for a sad facial expression image, calculating distance variations between eyebrows and the relatively fixed points, and distance variations between points near mouth corners and the relatively fixed points;

for a surprised facial expression image, calculating distance variations between eyebrows and lower eyelids, distance variations between an upper eyelid and a lower eyelid, and distance variations between an upper lip and a lower lip;

for an angry facial expression image, calculating distance variations between eyebrows and the relatively fixed points, and distance variations between eyebrows and eye centers; and

configuring differences between calculation results from synthetic images and extracted images as the plurality of key features;

wherein all the distance variations are measured relative to a neutral facial expression image; and

the dynamic features are obtained through steps of:

extracting facial key point coordinates from the plurality of facial expression images;

calculating positional relationships between upper and lower eyelids using extracted facial key point coordinates, and determining variation degrees of eye region key points through variance calculation;

calculating movement patterns of each key point in a mouth region using extracted facial key point coordinates, and determining variation degrees of the key points in the mouth region through variance calculation; and

configuring calculation results as the dynamic features.

2. An auxiliary diagnosis system for PD based on static and dynamic features of facial expressions, comprising:

a data acquisition module;

a data pro-processing module;

a static feature extraction module;

a dynamic feature extraction module; and

a classification prediction module;

wherein the data acquisition module is configured to acquire video data of various facial expressions performed by a to-be-tested patient;

the data pro-processing module is configured to pre-process the video data to extract a plurality of optimal facial expression images corresponding to the various facial expressions;

the static feature extraction module is configured to synthesize a happy facial expression image of the to-be-tested patient in a healthy state using a generative network to obtain a synthesized happy facial expression image and perform a similarity discrimination on the synthesized happy facial expression image and an extracted happy facial expression image based on a neutral facial expression image to obtain similarity features; and calculate distances between multiple facial key points in the various facial expression images to obtain a plurality of key features; and splice the similarity features and the plurality of key features to form static features;

the dynamic feature extraction module is configured to calculate coordinate change degrees of multiple facial key points of eyelids and mouth to obtain dynamic features based on the plurality of optimal facial expression images; and

the classification prediction module is configured to balance dimensions of the static features and the dynamic features, followed by feature splicing using a static-dynamic feature balanced classification network to obtain spliced features; and output a classification prediction result of PD based on the spliced features;

wherein the video data is pre-processed through steps of:

splitting the video into a plurality of independent videos, each of the plurality of independent videos corresponds to a facial expression;

annotating facial key points for consecutive video-frame images of each of the plurality of independent videos; convert coordinates of each facial key point to relative coordinates based on a relatively fixed point; and performing normalization processing on the video-frame images;

uniformly selecting K images from a neutral-expression independent video; and calculating an average value of coordinates of each facial key point in the K images to obtain key point coordinates of an average neutral face; and

for video-frame images in each expression-specific independent video, calculating a distance between each facial key point in each video-frame image and a key point in the average neutral face corresponding thereto; sorting video-frame images of each expression-specific independent video in a descending order according to a sum of distances of all key points in each video-frame image to screen the first L images as the optimal facial expression state images;

the various facial expressions comprise a neutral facial expression, a happy facial expression, a sad facial expression, a surprised facial expression, a fearful facial expression, an angry facial expression, and a disgusted facial expression; and

the facial key points comprise relatively fixed points and dynamic flexible points, the relatively fixed points comprise points around a nose;

the similarity discrimination is performed through steps of:

acquiring M synthetic happy facial expression images and N extracted happy facial expression images;

extracting coordinates of facial key points around a mouth from the acquired images; and

calculating Euclidean distance between the same facial key point in each synthetic image and each extracted image to obtain similarity discrimination results, and the discrimination results are configured as similarity features;

the plurality of key features are obtained through steps of:

for a happy facial expression image, calculating distance variations between points near mouth corners and the relatively fixed points, and distance changes between an upper lip and a lower lip;

for a sad facial expression image, calculating distance variations between eyebrows and the relatively fixed points, and distance variations between points near mouth corners and the relatively fixed points;

for a surprised facial expression image, calculating distance variations between eyebrows and lower eyelids, distance variations between an upper eyelid and a lower eyelid, and distance variations between an upper lip and a lower lip;

for an angry facial expression image, calculating distance variations between eyebrows and the relatively fixed points, and distance variations between eyebrows and eye centers; and

configuring differences between calculation results from synthetic images and extracted images as the plurality of key features; and

wherein all the distance variations are measured relative to a neutral facial expression image;

the dynamic features are obtained through steps of:

extracting facial key point coordinates from the plurality of facial expression images;

calculating positional relationships between upper and lower eyelids using extracted facial key point coordinates, and determining variation degrees of eye region key points through variance calculation;

calculating movement patterns of each key point in a mouth region using extracted facial key point coordinates, and determining variation degrees of the key points in the mouth region through variance calculation; and

configuring calculation results as the dynamic features.

3. An electronic device, comprising:

a memory; and

a processor;

wherein the memory is configured to store computer instructions; and the processor is configured to execute the computer instructions to implement the auxiliary diagnosis method of claim 1 .

4. A non-transitory computer-readable storage medium, wherein the computer-readable storage medium is configured to store computer instructions; and the computer instructions are configured to be executed by a processor to implement the auxiliary diagnosis method of claim 1 .

Assignments (1)

References Cited (11)