Determination method, determination apparatus of calibration information and electronic device

1 . A determination method of calibration information, comprising:

acquiring 3D key point coordinates and a pose of a target device in a Virtual Reality (VR) world coordinate system determined by a VR device, wherein the target device is associated with the VR device;

acquiring a target image collected by an image acquisition device, and acquiring 2D key point coordinates of the target device in a camera coordinate system based on the target image;

determining calibration information of the image acquisition device in the VR world coordinate system according to the 2D key point coordinates, the 3D key point coordinates, the pose of the target device and a pose of the image acquisition device, wherein the calibration information comprises position information and orientation information;

receiving a first screen image sent from the VR device after determining the calibration information of the image acquisition device in the VR world coordinate system;

acquiring a second screen image from a perspective of the image acquisition device based on the calibration information and the first screen image;

acquiring a real scene image of a user of the VR device collected by the image acquisition device after determining the calibration information of the image acquisition device in the VR world coordinate system;

acquiring a user screen image from the real scene image;

acquiring a prop screen image from the perspective of the image acquisition device, the prop screen image being a screen image of a prop associated with the target device; and

fusing the second screen image, the user screen image and the prop screen image to obtain a mixed reality capture (MRC) image.

2 . The determination method according to claim 1 , wherein the acquiring 3D key point coordinates and a pose of a target device in a VR world coordinate system determined by a VR device comprises:

receiving the 3D key point coordinates and the pose of the target device in the VR world coordinate system sent from the VR device.

3 . The determination method according to claim 1 , wherein the acquiring a target image collected by an image acquisition device comprises:

acquiring an original image of the target device collected by the image acquisition device; and

determining an image of a region of the target device from the original image, and cropping the image of the region of the target device from the original image as the target image.

4 . The determination method according to claim 1 , wherein the acquiring 2D key point coordinates of the target device in a camera coordinate system based on the target image comprises:

inputting the target image into a key point extraction model, to obtain 2D key point coordinates of the target device output by the key point extraction model,

wherein the key point extraction model is a convolutional neural network model trained based on sample information comprising a plurality of sample images of the target device and 2D key point coordinates corresponding to the target device of each sample image.

5 . The determination method according to claim 1 , further comprising:

determining at least one of a first layer relationship or a second layer relationship before fusing the second screen image, the user screen image and the prop screen image to obtain the MRC image, wherein the first layer relationship is a layer relationship between the user screen image and the second screen image, the second layer relationship is a layer relationship between the user screen image and the prop screen image, and at least one of the first layer relationship or the second layer relationship are used for fusing the second screen image, the user screen image and the prop screen image to obtain the MRC image.

6 . The determination method according to claim 5 , wherein the determining a first layer relationship comprises:

performing depth estimation on the real scene image to obtain first depth information of the real scene image;

acquiring second depth information of the second screen image; and

determining the first layer relationship between the user screen image and the second screen image according to the first depth information and the second depth information.

7 . The determination method according to claim 5 , wherein the determining a second layer relationship comprises:

determining the second layer relationship between the user screen image and the prop screen image in case that a target part of the user is detected to be visible in the real scene image,

wherein the target part is a part where the target device is worn or held.

8 . The determination method according to claim 1 , wherein the acquiring a user screen image from the real scene image comprises:

inputting the real scene image into a target matting model; and

acquiring a user screen image in the real scene image output by the target matting model,

wherein the target matting model is obtained through training based on a set of sample images comprising multiple composite images, each composite image being obtained by fusing a user image and an indoor scene image.

9 . The determination method according to claim 8 , further comprising:

acquiring a user green screen video before inputting the real scene image into the target matting model, the user green screen video being a user video collected in a green screen scene;

determining the user image of each frame of the user green screen video according to a full picture region of the user green screen video and a green screen region in the frame;

fusing the user image of each frame with an indoor scene image to obtain multiple composite images; and

determining a set of sample images based on the multiple composite images, and training an initial matting model based on the set of sample images to obtain the target matting model.

10 . The determination method according to claim 9 , wherein the determining the user image of each frame of the user green screen video according to a full picture region of the user green screen video and a green screen region in the frame comprises:

acquiring a green screen region in each frame of the user green screen video;

performing pixel completion on the green screen region of each frame to obtain a green screen completion region of the frame;

calculating an intersection of the green screen completion regions of the various frames to obtain a minimum green screen completion region;

removing the minimum green screen completion region from the full picture region of the user green screen video to obtain a green screen excluded region;

calculating the union of the green screen excluded region and a green screen extraction region of each frame to obtain a non-user image region of each frame, the green screen extraction region of each frame being a remaining green screen region after user image extraction based on the green screen region of each frame; and

determining the user image for each frame according to the full picture region and the non-user image region of the frame.

11 . The determination method according to claim 9 , wherein the fusing the user image of each frame with an indoor scene image to obtain multiple composite images comprises:

determining a first coordinate vector of a user positioning point of a target user image which is a user image of any one of the frames;

determining a second coordinate vector of a target positioning point in the indoor scene image;

calculating an offset vector according to the first coordinate vector and the second coordinate vector; and

fusing the target user image into the indoor scene image based on the offset vector to obtain a target composite image, wherein the user positioning point of the user image is located at the target positioning point in the indoor scene image.

12 . The determination method according to claim 11 , wherein:

the user positioning point is a center point of a bottom edge of a bounding rectangle of the target user image; and

the target positioning point is an arbitrary point in the indoor scene image.

13 . The determination method according to claim 12 , wherein:

the target user image comprises a user foot feature, and the target positioning point is a center point of a ground region in the indoor scene image; or

the target user image does not comprise a user foot feature, and the target positioning point is an arbitrary point on a bottom edge of the indoor scene image.

14 . The determination method according to claim 8 , wherein the target device is worn on or held by a target part of the user.

15 . The determination method according to claim 8 , wherein the VR device is worn by the user in the user image.

16 . A non-transitory computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the determination method of calibration information according to claim 1 .

17 . An electronic device, comprising:

a memory; and

a processor coupled to the memory, the processor configured to execute the determination method of calibration information for performing instructions comprising:

acquiring 3D key point coordinates and a pose of a target device in a Virtual Reality (VR) world coordinate system determined by a VR device, wherein the target device is associated with the VR device;

acquiring a target image collected by an image acquisition device, and acquiring 2D key point coordinates of the target device in a camera coordinate system based on the target image;

determining calibration information of the image acquisition device in the VR world coordinate system according to the 2D key point coordinates, the 3D key point coordinates, the pose of the target device and a pose of the image acquisition device, wherein the calibration information comprises position information and orientation information;

receiving a first screen image sent from the VR device after determining the calibration information of the image acquisition device in the VR world coordinate system;

acquiring a second screen image from a perspective of the image acquisition device based on the calibration information and the first screen image;

acquiring a real scene image of a user of the VR device collected by the image acquisition device after determining the calibration information of the image acquisition device in the VR world coordinate system;

acquiring a user screen image from the real scene image;

acquiring a prop screen image from the perspective of the image acquisition device, the prop screen image being a screen image of a prop associated with the target device; and

fusing the second screen image, the user screen image and the prop screen image to obtain a mixed reality capture (MRC) image.