Systems and methods for fusing EEG and fMRI through hierarchical deep transcoding

1. A system for hierarchical deep transcoding, comprising:

a processor configured to

receive a functional magnetic resonance imaging (fMRI) data and/or an extracranial electroencephalogram (EEG) data, wherein the fMRI data and the EEG data are simultaneously acquired; and

reconstruct a latent source space from the fMRI data and/or the EEG data by decoding the EEG data and/or the fMRI data to the latent source space, wherein the processor is configured to transcode the EEG data to an fMRI signal or the fMRI data to an EEG signal.

2. The system of claim 1 , wherein the processor is further configured to

train a group-level model with a plurality of EEG and fMRI signals obtained from at least two subjects to determine an intermediate spatial/temporal resolution of the plurality of EEG and fMRI signals; and

generate estimates of the latent source space of the intermediate solution from the EEG and fMRI signals based on the trained group-level model, wherein the interpolated EEG and fMRI signals have a same intermediate resolution for training the group-level model.

3. The system of claim 2 , wherein the processor is configured to

train a subject-level model with the group-level latent space estimates from the EEG and fMRI signals to determine a spatial/temporal resolution of the subject-level latent source space, wherein the group-level latent space estimates are epoched to achieve a 3D event related potential (ERP) representation; and

determine the spatial/temporal resolution of the latent source space using the subject-level model.

4. The system of claim 3 , wherein the epoched signals are sliced in a spatial and temporal direction.

5. The system of claim 4 , wherein the group-level model and the subject-level model are a cyclic convolutional transcoder.

6. The system of claim 4 , wherein the EEG data is transcoded to an fMRI signal and/or the fMRI data is transcoded to an EEG signal based on the spatial/temporal resolution of the latent source space determined by the group-level model and the subject-level model.

7. The system of claim 5 , wherein the cyclic convolutional transcoder comprises an fMRI decoder, an fMRI encoder, an EEG decoder, an EEG encoder, or combinations thereof.

8. The system of claim 7 , wherein the fMRI decoder decodes the latent source space from an fMRI encoding, and the EEG decoder decodes the latent source space from an EEG encoding.

9. The system of claim 7 , wherein the fMRI encoder encodes the latent source space into an fMRI encoding, and the EEG encoder encodes the latent source space into an EEG encoding.

10. The system of claim 7 , wherein the fMRI decoder and the fMRI encoder comprise temporal convolutional layers.

11. The system of claim 7 , wherein the EEG decoder and the EGG encoder comprise spatial convolutional layers.

12. The system of claim 1 , wherein the processor is configured to generate a map based on the reconstructed the EEG signal and/or the fMRI signal.

13. The system of claim 1 , wherein the processor is configured to estimate a hemodynamic impulse response (RF) function; and

forward or inverse a head conductivity model from the fMRI and EEG data, wherein the conductivity model includes a leadfield matrix.

14. A method for hierarchical deep transcoding, comprising

receiving a functional magnetic resonance imaging (fMRI) data and/or an extracranial electroencephalogram (EEG) data, wherein the fMRI data and the EEG data are simultaneously acquired;

reconstructing a latent source space from the EEG data and/or the fMRI data by decoding the EEG data and the fMRI data to the latent source space;

training a group-level model with a plurality of EEG and fMRI signals obtained from at least two subjects to determine an intermediate spatial/temporal resolution of the plurality of EEG and fMRI signals; and

generating estimates of the latent source space of the intermediate resolution from EEG and fMRI signals based on the trained group-level model, wherein the interpolated EEG and fMRI signals have a same intermediate resolution for training the group-level model.

15. The method of claim 5 , further comprising

training a subject-level model with the group-level latent space estimates from the EEG and fMRI signals to determine a spatial/temporal resolution of the latent source space, wherein the ground-level latent space estimates are epoched to achieve a 3D event related potential (ERP) representation; and

determining the spatial/temporal resolution of the latent source space using the subject-level model.

16. The method of claim 15 , wherein the EEG data is transcoded to an fMRI signal or the fMRI data is transcoded to an EEG signal based on the spatial/temporal resolution of the latent source space determined by the group-level model and the subject-level model.

17. The method of claim 16 , further comprising generating a map based on the reconstructed the EEG signal and/or the fMRI signal, wherein the map comprises an activation map, a Z-statistic map, or combinations thereof.

18. The method of claim 14 , further comprising

estimating a hemodynamic impulse response (RF) function; and

forwarding or inversing a head a conductivity model from the fMRI and EEG data, wherein the conductivity model includes a leadfield matrix.