Multi-dimensional fish school perception apparatus and method

1 . A multi-dimensional fish school perception method, comprising:

acquiring a multi-view optical image and original sonar data of a fish school by using a multi-dimensional fish school perception apparatus;

obtaining a first three-dimensional (3D) feature map of the fish school based on the multi-view optical image;

obtaining a second 3D feature map of the fish school based on the original sonar data;

extracting fused features based on the first 3D feature map and the second 3D feature map to obtain a fused feature map; and

obtaining, based on the fused feature map, a 3D detection result of a target fish school by using a self-attention-mechanism-based 3D target detection method;

wherein obtaining the first 3D feature map of the fish school based on the multi-view optical image comprises:

inputting the multi-view optical image into a first convolutional neural network to obtain feature codes of the multi-view optical image;

obtaining, based on the feature codes, a fused two-dimensional (2D) feature by using an adaptive-feature-pyramid-based multi-scale feature fusion method; and

based on the 2D feature and information of intrinsic parameters and dynamic extrinsic parameters of a plurality of pan-tilt cameras, using a spatial projection method to complete depth prediction and obtain the first 3D feature map;

wherein the multi-dimensional fish school perception apparatus comprises a fixed workbench, the plurality of pan-tilt cameras, and an omnidirectional scanning sonar,

wherein the omnidirectional scanning sonar is located at a central position of the fixed workbench and configured to acquire point cloud information in a 360-degree range around the multi-dimensional fish school perception apparatus;

the plurality of pan-tilt cameras are uniformly distributed on the fixed workbench, and have a same distance from the central position; the plurality of pan-tilt cameras are configured to rotate with one degree of freedom in a horizontal direction; and the plurality of pan-tilt cameras are configured to actively rotate according to distribution of optical image feature points in an environment; and

the plurality of pan-tilt cameras are configured to acquire the multi-view optical image.

2 . The multi-dimensional fish school perception method according to claim 1 , wherein acquiring the multi-view optical image and the original sonar data of the fish school comprises: performing collection of the multi-view optical image and the original sonar data in an underwater environment by using an underwater robot equipped with the multi-dimensional fish school perception apparatus to acquire the multi-view optical image and the original sonar data.

3 . The multi-dimensional fish school perception method according to claim 1 , wherein obtaining the second 3D feature map of the fish school based on the original sonar data comprises:

correcting the original sonar data to obtain point cloud data with strengthened time alignment degree;

performing voxel transformation on the point cloud data to obtain voxel data; and

obtaining, based on the voxel data, the second 3D feature map by using a sparse 3D convolutional neural network.

4 . The multi-dimensional fish school perception method according to claim 3 , wherein correcting the original sonar data to obtain the point cloud data with the strengthened time alignment degree comprises:

performing filtering, denoising, correcting and normalization processing on the original sonar data to obtain preprocessed sonar data;

predicting a delay value of the original sonar data by using an auto-regressive moving average model; and

correcting the preprocessed sonar data based on the delay value to obtain the point cloud data with the strengthened time alignment degree.

5 . The multi-dimensional fish school perception method according to claim 1 , wherein extracting the fused features based on the first 3D feature map and the second 3D feature map comprises: cascading data of the first 3D feature map and the second 3D feature map according to channels, and further extracting the fused features by using a second convolutional neural network.

6 . The multi-dimensional fish school perception method according to claim 1 , wherein obtaining, based on the fused feature map, the 3D detection result of the target fish school by using the self-attention-mechanism-based 3D target detection method comprises:

based on the fused feature map at a current moment, refining through multi-layer convolution, global pooling and a fully-connected layer to obtain a spatial feature within the fused features;

obtaining an optical flow feature based on a displacement vector field by using a continuous fused feature map sequence;

based on the optical flow feature, refining through multi-layer convolution, global pooling and the fully-connected layer to obtain a temporal feature within the fused features; and

merging the temporal feature and the spatial feature layer by layer to obtain a merged feature, and inputting the merged feature into a feature fusion layer to obtain a behavior detection result of the target fish school;

wherein the spatial feature comprises a relative position of the fish school and scenario information; and

the temporal feature comprises position changes of the fish school at different moments, and the temporal feature is used to reflect movement information of the fish school.

7 . A multi-dimensional fish school perception system, comprising:

an acquisition module, configured to acquire a multi-view optical image and original sonar data of a fish school by using a multi-dimensional fish school perception apparatus;

a first feature extraction module, configured to obtain a first 3D feature map of the fish school based on the multi-view optical image;

a second feature extraction module, configured to obtain a second 3D feature map of the fish school based on the original sonar data;

a feature fusion module, configured to extract fused features based on the first 3D feature map and the second 3D feature map to obtain a fused feature map; and

a result detection module, configured to obtain, based on the fused feature map, a 3D detection result of a target fish school by using a self-attention-mechanism-based 3D target detection method;

wherein obtaining the first 3D feature map of the fish school based on the multi-view optical image comprises:

inputting the multi-view optical image into a first convolutional neural network to obtain feature codes of the multi-view optical image;

obtaining, based on the feature codes, a fused 2D feature by using an adaptive-feature-pyramid-based multi-scale feature fusion method; and

based on the 2D feature and information of intrinsic parameters and dynamic extrinsic parameters of a plurality of pan-tilt cameras, using a spatial projection method to complete depth prediction and obtain the first 3D feature map;

wherein the multi-dimensional fish school perception apparatus comprises a fixed workbench, the plurality of pan-tilt cameras, and an omnidirectional scanning sonar,

wherein the omnidirectional scanning sonar is located at a central position of the fixed workbench and configured to acquire point cloud information in a 360-degree range around the multi-dimensional fish school perception apparatus;

the plurality of pan-tilt cameras are uniformly distributed on the fixed workbench, and have a same distance from the central position; the plurality of pan-tilt cameras are configured to rotate with one degree of freedom in a horizontal direction; and the plurality of pan-tilt cameras are configured to actively rotate according to distribution of optical image feature points in an environment; and

the plurality of pan-tilt cameras are configured to acquire the multi-view optical image.

8 . An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, is configured to implement the multi-dimensional fish school perception method according to claim 1 .

9 . The electronic device according to claim 8 , wherein acquiring the multi-view optical image and the original sonar data of the fish school comprises: performing collection of the multi-view optical image and the original sonar data in an underwater environment by using an underwater robot equipped with the multi-dimensional fish school perception apparatus to acquire the multi-view optical image and the original sonar data.

10 . The electronic device according to claim 8 , wherein obtaining the second 3D feature map of the fish school based on the original sonar data comprises:

correcting the original sonar data to obtain point cloud data with strengthened time alignment degree;

performing voxel transformation on the point cloud data to obtain voxel data; and

obtaining, based on the voxel data, the second 3D feature map by using a sparse 3D convolutional neural network.

11 . The electronic device according to claim 10 , wherein correcting the original sonar data to obtain the point cloud data with the strengthened time alignment degree comprises:

performing filtering, denoising, correcting and normalization processing on the original sonar data to obtain preprocessed sonar data;

predicting a delay value of the original sonar data by using an auto-regressive moving average model; and

correcting the preprocessed sonar data based on the delay value to obtain the point cloud data with the strengthened time alignment degree.

12 . The electronic device according to claim 8 , wherein extracting the fused features based on the first 3D feature map and the second 3D feature map comprises: cascading data of the first 3D feature map and the second 3D feature map according to channels, and further extracting the fused features by using a second convolutional neural network.

13 . The electronic device according to claim 8 , wherein obtaining, based on the fused feature map, the 3D detection result of the target fish school by using the self-attention-mechanism-based 3D target detection method comprises:

based on the fused feature map at a current moment, refining through multi-layer convolution, global pooling and a fully-connected layer to obtain a spatial feature within the fused features;

obtaining an optical flow feature based on a displacement vector field by using a continuous fused feature map sequence;

based on the optical flow feature, refining through multi-layer convolution, global pooling and the fully-connected layer to obtain a temporal feature within the fused features; and

merging the temporal feature and the spatial feature layer by layer to obtain a merged feature, and inputting the merged feature into a feature fusion layer to obtain a behavior detection result of the target fish school;

wherein the spatial feature comprises a relative position of the fish school and scenario information; and

the temporal feature comprises position changes of the fish school at different moments, and the temporal feature is used to reflect movement information of the fish school.