Method and system for early diagnosis of Parkinson's disease based on multimodal deep learning

1 . A method for early diagnosis of Parkinson's disease based on multimodal deep learning, comprising:

(1) acquiring audio-visual data of a to-be-diagnosed subject while performing a speech task;

(2) preprocessing the audio-visual data to extract a plurality of audio segments and a plurality of video segments, wherein each of the plurality of audio segments corresponds to a synchronized one among the plurality of video segments; extracting a face image sequence from each of the plurality of video segments; and calculating a Mel-spectrogram of each of the plurality of audio segments; and

(3) inputting the face image sequence and the Mel-spectrogram into a multimodal deep learning model to output a classification result for Parkinson's disease early diagnosis of the to-be-diagnosed subject, wherein the multimodal deep learning model comprises a local feature extraction module, an audio feature extraction module, a feedforward network and a cross-attention module;

wherein step (3) is performed through steps of:

(3.1) extracting visual features from the face image sequence through the local feature extraction module, and extracting audio features from the Mel-spectrogram through the audio feature extraction module; and

(3.2) inputting the visual features and the audio features to the feedforward network, and inputting the visual features and the audio features to the cross-attention module to learn a cross-modal attention weight; performing feature fusion on the visual features and the audio features based on the cross-modal attention weight to obtain multimodal features; and outputting the classification result based on the multimodal features.

2 . The method of claim 1 , wherein the local feature extraction module comprises a visual front-end network and a visual temporal network;

the visual front-end network is based on ShuffleNet-V2, and further comprises a two-dimensional (2D) convolution module; the visual front-end network is configured to encode the face image sequence into a frame-based embedding sequence; and the visual temporal network consists of a video temporal convolution module, and is configured to capture facial motion visual features in different time intervals; and

the step of extracting the visual features from the face image sequence through the local feature extraction module comprises:

extracting facial visual features from each frame of the face image sequence through the visual front-end network, and extracting the visual features from the facial visual features through the visual temporal network, wherein the visual features are time-correlated.

3 . The method of claim 1 , wherein the audio feature extraction module is a VGGish network provided with a convolution module; the audio feature extraction module is configured to extract the audio features at different time intervals from the plurality of audio segments; and

the step of extracting the audio features from the Mel-spectrogram through the audio feature extraction module comprises:

inputting the Mel-spectrogram into the audio feature extraction module, and extracting the audio features through the VGGish network, wherein the audio features are time-correlated.

4 . The method of claim 1 , wherein step (3.2) comprises:

after the visual features and the audio features pass through the feedforward network, inputting the visual features and the audio features into the cross-attention module with the visual features as key vectors and value vectors and the audio features as query vectors to learn the cross-modal attention weight, and acquiring visual feature-enhanced audio features based on the cross-modal attention weight; and inputting the visual features and the audio features into the cross-attention module with the audio features as the key vectors and the value vectors and the visual features as the query vectors to learn the cross-modal attention weight, and acquiring audio feature-enhanced visual features based on the cross-modal attention weight; and

fusing the visual feature-enhanced audio features with the audio features to obtain first fused features, and fusing the audio feature-enhanced visual features with the visual features to obtain second fused features, and concatenating the first fused features with the second fused features to obtain the multimodal features.

5 . The method of claim 1 , wherein the multimodal deep learning model is trained through steps of:

collecting a plurality of sets of audio-visual data of a plurality of test subjects while performing the speech task, wherein the plurality of test subjects comprise Parkinson's disease patients and healthy subjects; performing disease severity evaluation according to a unified Parkinson's disease rating scale (UPDRS) to annotate and score the plurality of sets of audio-visual data; and constructing a training data set based on the plurality of sets of annotated audio-visual data; and

based on the training data set, training the multimodal deep learning model by means of a cross-entropy loss and a stochastic gradient descent optimizer until a preset number of iterations is reached.

6 . A non-transitory computer-readable storage medium, wherein a computer instruction is stored on the non-transitory computer-readable storage medium; and the computer instruction is configured to be executed by a processor to implement the method of claim 1 .