Method and apparatus for constructing three-dimensional data set of pedestrian re-identification based on neural radiation field

1. A method for constructing a three-dimensional data set of a pedestrian re-identification based on a neural radiation field, comprising the following steps:

S1: capturing images of pedestrians to be entered by a group of cameras at different viewing angles;

S2: generating a three-dimensional spatial position point set by sampling through camera rays in the scenario, and converting observation directions of the cameras corresponding to the three-dimensional spatial position point set into three-dimensional Cartesian unit vectors;

S3: inputting, into a multi-layer sensor, the three-dimensional spatial position point set and the observation directions converted into the three-dimensional Cartesian unit vectors, to output corresponding densities and colors;

S4: accumulating, by using a neural volume rendering method, ray colors passing through each pixel into the images captured in step S1, including the following sub-steps:

S4-1: defining cumulative transparency rates of the camera rays by continuous integration, and generating definitions of the ray colors accordingly;

S4-2: estimating the ray colors by using a quadrature method, dividing near boundaries to far boundaries of the rays into N uniformly spaced intervals, and selecting discrete points by using a stratified sampling method;

S5: introducing position encoding and multi-level sampling to improve the quality of the images generated by the accumulation of ray colors in step S4, specifically:

S5-1: introducing position encoding: encoding the spatial positions of the points, and converting the three-dimensional vectors input into a neural network into specified dimensions, to increase the accuracy of the generated images;

S5-2: introducing multi-level sampling: first collecting a group of points by stratified sampling, preliminarily evaluating the neural network, generating a probability density function based on the output of the preliminarily evaluated neural network, then collecting points along each ray based on the probability density function, and combining the points sampled twice to evaluate the neural network more accurately; and

S6: labeling the generated images and storing the same in a data set.

2. The method for constructing a three-dimensional data set of a pedestrian re-identification based on a neural radiation field according to claim 1 , wherein the three-dimensional spatial position point set in step S2 refers to three-dimensional spatial positions (x, y, z) where the cameras are located, and the observation directions of the cameras corresponding to the three-dimensional spatial position point set are d, which can be converted into the three-dimensional Cartesian unit vectors.

3. The method for constructing a three-dimensional data set of a pedestrian re-identification based on a neural radiation field according to claim 1 , wherein the specific process of step S3 is: inputting, into the multi-layer sensor, the spatial positions and observation directions (ζ, d) of the cameras, to output colors and densities (c, σ) of points, where ζ is the spatial position (x, y, z), d is the three-dimensional Cartesian unit vector converted from the observation direction, c is the color, and σ is the volume density.

4. The method for constructing a three-dimensional data set of a pedestrian re-identification based on a neural radiation field according to claim 1 , wherein the neural volume rendering method in step S4 is specifically as follows: tracing light in the scenario, and integrating specified lengths of light to generate images or videos, wherein in the method of generating images from three-dimensional scalar data, the color of any ray passing through the scenario is rendered into an image.

5. The method for constructing a three-dimensional data set of a pedestrian re-identification based on a neural radiation field according to claim 1 , wherein the specific process of step S4-1 is: marking each camera ray as r(t)=o+td, where o is an origin of the ray, d is the observation direction, t refers to the position of a certain point in the space through which the camera ray passes, and the specific definition of a ray color is as follows:

C ( r )=∫ t n t f T ( t )σ( r ( t )) c ( r ( t ), d ) dt;

wherein t n and t f are the near boundary and far boundary of the ray, c is the color, σ is the volume density, and T(t) is a cumulative transparency of the ray on the path from t n to t, that is, a probability that the ray on the path from t n to t does not hit any particle, specifically:

T ( t )=exp(−∫ t n t σ( r ( s ) ds ).

6. The method for constructing a three-dimensional data set of a pedestrian re-identification based on a neural radiation field according to claim 5 , wherein step S4-2 is specifically: dividing the distance [t n , t f ] between the near boundary t n and far boundary t f of the ray into N uniformly spaced intervals, and then randomly extracting a sample from each interval, that is, t i obeys a uniform distribution:

t

i

∼

U

[

t

n

+

i

-

1

N

⁢

(

t

f

-

t

n

)

,

t

n

+

i

-

1

N

⁢

(

t

f

-

t

n

)

]

,

i

=

1

,

2

,

…

,

N

;

then the integration formula of the ray colors C(r) can be simplified as:

C

^

(

r

)

=

∑

i

=

1

N

T

i

·

(

1

-

exp

⁡

(

-

σ

i

⁢

δ

i

)

)

·

c

i

;

wherein

δ i =t i+1 −t i ,

T i =exp(−Σ j=1 i−1 σ j δ j ), σ i represents the volume density, and c i represents the color.

7. The method for constructing a three-dimensional data set of a pedestrian re-identification based on a neural radiation field according to claim 1 , wherein the specific method of introducing position encoding in step S5-1 is: standardizing the spatial positions ζ and the observation directions d, and encoding each coordinate value in the spatial positions and the observation directions as follows:

γ( p )=(sin( 2 0 πp ),cos( 2 0 πp ), . . . , sin( 2 L−1 πp ),cos ( 2 L−1 πp )).

8. The method for constructing a three-dimensional data set of a pedestrian re-identification based on a neural radiation field according to claim 6 , wherein the specific sub-steps of introducing multi-level sampling in step S5-2 are as follows:

step 1: collecting N c points on the rays by stratified sampling;

step 2: inputting the sampled points, and preliminarily evaluating the neural network at the sampled points by using the quadrature method;

step 3: generating the probability density function by normalization, and rewriting the integration formula in step S4-2 as:

C

r

^

(

r

)

=

∑

i

=

1

N

ω

i

⁢

c

i

;

wherein

ω i =T i ·(1−exp(−σ i ·δ i ), ω i is normalized

ω

^

i

=

ω

i

∑

j

=

1

x

c

⁢

ω

j

to generate a piecewise constant probability density function;

step 4: collecting N f points along each ray based on the above probability density function; and

step 5: evaluating the neural network more accurately by using the collected (N c +N f ) points, so as to better render the ray colors.

9. A computer program product comprising a non-transitory computer readable medium having instructions recorded thereon, the instructions when executed by a computer implementing a method for constructing a three-dimensional data set of a pedestrian re-identification based on a neural radiation field, comprising the following steps:

S1: capturing images of pedestrians to be entered by a group of cameras at different viewing angles;

S2: generating a three-dimensional spatial position point set by sampling through camera rays in the scenario, and converting observation directions of the cameras corresponding to the three-dimensional spatial position point set into three-dimensional Cartesian unit vectors;

S3: inputting, into a multi-layer sensor, the three-dimensional spatial position point set and the observation directions converted into the three-dimensional Cartesian unit vectors, to output corresponding densities and colors;

S4: accumulating, by using a neural volume rendering method, ray colors passing through each pixel into the images captured in step S1, including the following sub-steps:

S4-1: defining cumulative transparency rates of the camera rays by continuous integration, and generating definitions of the ray colors accordingly;

S4-2: estimating the ray colors by using a quadrature method, dividing near boundaries to far boundaries of the rays into N uniformly spaced intervals, and selecting discrete points by using a stratified sampling method;

S5: introducing position encoding and multi-level sampling to improve the quality of the images generated by the accumulation of ray colors in step S4, specifically:

S5-1: introducing position encoding: encoding the spatial positions of the points, and converting the three-dimensional vectors input into a neural network into specified dimensions, to increase the accuracy of the generated images;

S5-2: introducing multi-level sampling: first collecting a group of points by stratified sampling, preliminarily evaluating the neural network, generating a probability density function based on the output of the preliminarily evaluated neural network, then collecting points along each ray based on the probability density function, and combining the points sampled twice to evaluate the neural network more accurately; and

S6: labeling the generated images and storing the same in a data set.