Panoramic video processing method and device and non-transitory computer-readable medium

1. A panoramic video processing method comprising:

receiving a panoramic video to be processed, the panoramic video including a plurality of two dimensional (2D) panoramic images;

conducting inter frame motion estimation with respect to each two neighboring 2D panoramic images in the panoramic video so as to obtain a corresponding inter frame motion estimation result between the same two neighboring 2D panoramic images;

calculating, based on the respective inter frame motion estimation results obtained between the plurality of 2D panoramic images, a historical motion trajectory of the panoramic video, and carrying out smoothing with respect to the historical motion trajectory;

calculating, based on the historical motion trajectory after smoothing, a rotation matrix for motion compensation pertaining to each 2D panoramic image in the panoramic video, the calculating including processes of

setting a first 2D panoramic image in the plurality of 2D panoramic images as a first reference 2D panoramic image; and

determining, based on the historical motion trajectory after smoothing, whether a change in a current 2D panoramic image relative to a previous 2D panoramic image is due to unconscious noise motion or conscious camera motion, wherein

in a case where the change is determined as unconscious noise motion, the rotation matrix for motion compensation of the current 2D panoramic image, relative to the previous 2D panoramic image, is calculated, and

in a case where the change is determined as conscious camera motion, the first reference 2D panoramic image is replaced with a second reference 2D panoramic image from a later time than the first 2D panoramic image, and the rotation matrix for motion compensation is then recalculated based on the second reference 2D panoramic image and the current 2D panoramic image; and

performing reconstruction on each 2D panoramic image in the panoramic video by way of the corresponding rotation matrix for motion compensation so as to attain a panoramic video after image stabilization, and

outputting the panoramic video after image stabilization.

2. The panoramic video processing method according to claim 1 , wherein, the conducting inter frame motion estimation with respect to each two neighboring 2D panoramic images in the panoramic video so as to obtain an inter frame motion estimation result between the same two neighboring 2D panoramic images includes

carrying out a matching process with respect to feature points in an N+1-th 2D panoramic image and feature points in an N-th 2D panoramic image so as to gain matching pairs, N being a positive integer; and

determining correct matching pairs from all the matching pairs, and computing, based on the correct matching pairs, a rotation matrix from the N+1-th 2D panoramic image to the N-th panoramic image.

3. The panoramic video processing method according to claim 2 , wherein, the determining correct matching pairs from all the matching pairs includes

selecting M matching pairs from all the matching pairs, M being a positive integer greater than one;

calculating, based on the M matching pairs, a first rotation matrix from the N+1-th 2D panoramic image to the N-th 2D panoramic image;

computing, by way of the first rotation matrix, errors of other matching pairs except the M matching pairs in all the matching pairs, and determining, based on the respective errors computed, a number of matching pairs supporting the first rotation matrix; and

determining, if the number of matching pairs supporting the first rotation matrix is greater than a predetermined threshold, that the M matching pairs are correct matching pairs, otherwise, that the M matching pairs are not correct matching pairs.

4. The panoramic video processing method according to claim 1 , wherein, the determining, based on the historical motion trajectory after smoothing, that a change of a current 2D panoramic image relative to a previous 2D panoramic image is unconscious noise motion or conscious camera motion includes

calculating, based on the historical motion trajectory, a motion noise value of the current 2D panoramic image relative to the previous 2D panoramic image; and

determining, if the motion noise value is less than or equal to a predetermined threshold, that the change is unconscious noise motion including unconscious noise motion within a predetermined noise tolerance range and unconscious noise motion exceeding the predetermined noise tolerance range, and if the motion noise value is greater than the predetermined threshold, that the change is conscious camera motion, the predetermined noise tolerance range being less than or equal to another threshold which is less than the predetermined threshold.

5. A panoramic video processing device comprising:

a processor; and

a memory storing program instructions that cause the processor to:

receive a panoramic video to be processed, the panoramic video including a plurality of two dimensional (2D) panoramic images;

conduct inter frame motion estimation in regard to each two neighboring 2D panoramic images in the panoramic video so as to obtain a corresponding inter frame motion estimation result between the same two neighboring 2D panoramic images;

calculate, based on the respective inter frame motion estimation results obtained between the plurality of 2D panoramic images, a historical motion trajectory of the panoramic video, and carrying out smoothing with respect to the historical motion trajectory;

calculate, based on the historical motion trajectory after smoothing, a rotation matrix for motion compensation pertaining to each 2D panoramic image in the panoramic video, the calculating including processes of

setting a first 2D panoramic image in the plurality of 2D panoramic images as a first reference 2D panoramic image; and

determining, based on the historical motion trajectory after smoothing, whether a change in a current 2D panoramic image relative to a previous 2D panoramic image is due to unconscious noise motion or conscious camera motion, wherein

in a case where the change is determined as unconscious noise motion, the rotation matrix for motion compensation of the current 2D panoramic image, relative to the previous 2D panoramic image, is calculated, and

in a case where the change is determined as conscious camera motion, the first reference 2D panoramic image is replaced with a second reference 2D panoramic image from a later time than the first 2D panoramic image, and the rotation matrix for motion compensation is then recalculated based on the second reference 2D panoramic image and the current 2D panoramic image; and

reconstruct on each 2D panoramic image in the panoramic video by way of the corresponding rotation matrix for motion compensation so as to attain a panoramic video after image stabilization, and to output the panoramic video after image stabilization.

6. A non-transitory computer-readable medium having computer-executable instructions for execution by a processing system, wherein, the computer-executable instructions, when executed, cause the processing system to carry out the panoramic video processing method according to claim 1 .