IP Library › Granted Patent US 11,037,364

Granted Patent B2

US 11,037,364 · App. 16/372,482 · Granted Jun 15, 2021

Image processing system for generating a virtual viewpoint image, method of controlling image processing system, and storage medium

Inventors: Akihiko Yushiya (Tokyo, JP); Yuya Ota (Kawasaki, JP)

Assignee: CANON KABUSHIKI KAISHA

G06T15/205 G06F3/04815 G06T1/00 G06T1/0007 G06T11/80 G06T19/00 H04N5/225 H04N5/232 H04N5/23229 H04N5/247

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,037,364
App. No.: 16/372,482
Granted: Jun 15, 2021
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

An image processing system, which generates a virtual viewpoint image on the basis of captured images from a plurality of image capturing apparatuses for capturing an object from a plurality of directions, detects an error in at least one of the plurality of image capturing apparatuses, and generates the virtual viewpoint image using a captured image from an image capturing apparatus, among the plurality of image capturing apparatuses, in which an error has not been detected, as an image of a capturing field of the image capturing apparatus in which the error has been detected.

Claims (48)

1. An image processing system comprising:

a plurality of image processing apparatuses including an image processing apparatus that

(i) transmits, to a first another image processing apparatus among the plurality of image processing apparatuses, an image that is derived from at least one of a plurality of image capturing apparatuses and is used for generating a virtual viewpoint image,

(ii) receives an image transmitted from a second another image processing apparatus among the plurality of image processing apparatuses different from the first another image processing apparatus,

(iii) transmits, in a case where an image to be received from the second another image processing apparatus is received, the received image to the first another image processing apparatus, and

(iv) transmits, in a case where an image to be received from the second another image processing apparatus is not received, an alternate image to the first another image processing apparatus, the alternate image being an image used for the virtual viewpoint image; and

a generating apparatus that generates the virtual viewpoint image based on a plurality of images transmitted by the plurality of image processing apparatuses.

2. The image processing system according to claim 1 , wherein

the plurality of image capturing apparatuses are connected to the plurality of image processing apparatuses;

the image processing apparatus generates an image for transmitting to the first another image processing apparatus based on a captured image obtained from a connected image capturing apparatus among the plurality of image capturing apparatuses; and

in the case where an image to be received from the second another image processing apparatus is not received, the image processing apparatus generates the alternate image for transmitting to the first another image processing apparatus.

3. The image processing system according to claim 1 , wherein

in a case where an image to be received from the second another image processing apparatus is not received before a predetermined amount of time passes, the image processing apparatus transmits the alternate image to the first another image processing apparatus.

4. The image processing system according to claim 1 , wherein

the plurality of image processing apparatuses are daisy-chained.

5. The image processing system according to claim 1 , wherein in a case where a background image, which does not include a region corresponding to a specified object, to be received from the second another image processing apparatus is not received, the image processing apparatus transmits an alternate background image to the first another image processing apparatus.

6. The image processing system according to claim 1 , wherein:

the image processing apparatus generates an image used for generating the virtual viewpoint image by extracting a partial image from a captured image obtained from the connected image capturing apparatus; and

in the case where an image to be received from the second another image processing apparatus is not received, the image processing apparatus generates the alternate image by changing an extraction range of the partial image extracted from the captured image.

7. The image processing system according to claim 6 , wherein

the image processing apparatus changes a capturing condition of the connected image capturing apparatus so that a region including at least part of a region corresponding to the partial image to be received from the second another image processing apparatus is able to be extracted from the captured image obtained by the connected image capturing apparatus.

8. The image processing system according to claim 7 , wherein

the capturing condition includes an angle of view and/or a capturing direction of an image capturing apparatus.

9. The image processing system according to claim 6 , wherein

the image processing apparatus changes the extraction range so that a region corresponding to the extracted partial image includes at least part of a region corresponding to the partial image to be received from the second another image capturing apparatus.

10. The image processing system according to claim 6 , wherein

the image processing apparatus changes the extraction range so that a region corresponding to a partial image for transmitting to the first another image processing apparatus and a region corresponding to a partial image to be received from the second another image processing apparatus are extracted individually from a captured image obtained from the connected image capturing apparatus.

11. The image processing system according to claim 10 , wherein

a partial image including a extracted region corresponding to the partial image to be received from the second another image processing apparatus is transformed based on capturing conditions of the connected image capturing apparatus and an image capturing apparatus connected to the second another image processing apparatus, and is transmitted to the first another image processing apparatus.

12. A method of controlling an image processing system comprising: a plurality of image processing apparatuses including an image processing apparatus, a first another image processing apparatus, and a second another image processing apparatus different from the first another image processing apparatus; and a generating apparatus, the method comprising:

transmitting, by the image processing apparatus, to the first another image processing apparatus an image that is derived from at least one of a plurality of image capturing apparatuses and is used for generating a virtual viewpoint image;

receiving, by the image processing apparatus, an image transmitted from the second another image processing apparatus;

transmitting, in a case where an image to be received from the second another image processing apparatus is received, the received image to the first another image processing apparatus;

transmitting, in a case where an image to be received from the second another image processing apparatus is not received, an alternate image to the first another image processing apparatus, the alternate image being an image used for the virtual viewpoint image; and

generating, by the generating apparatus, the virtual viewpoint image based on a plurality of images transmitted by the plurality of image processing apparatuses.

13. A non-transitory computer-readable storage medium storing a computer program for causing a computer to execute a method of controlling an image processing apparatus constituting a plurality of image processing apparatus that transmit a plurality of images to an image generating apparatus which generates a virtual viewpoint image on the basis of the plurality of images, the method comprising:

transmitting, to a first another image processing apparatus among the plurality of image processing apparatuses, an image that is derived from at least one of a plurality of image capturing apparatuses and is used for generating a virtual viewpoint image;

receiving an image transmitted from a second another image processing apparatus among the plurality of image processing apparatuses different from the first another image processing apparatus;

transmitting, in a case where an image to be received from the second another image processing apparatus is received, the received image to the first another image processing apparatus; and

14. The image processing system according to claim 1 , wherein

in a case where an image is not received from the second another image processing apparatus before a predetermined amount of time passes, the image processing apparatus generates and transmits the alternate image to the first another image processing apparatus.

15. The image processing system according to claim 1 , wherein

in a case where the image received from the second another image processing apparatus is based on an erroneous image capturing by an image capturing apparatus, the image processing apparatus transmits the alternate image to the first another image processing apparatus.

16. The image processing system according to claim 1 , wherein

the image processing apparatus detects an error related to the image received from the second another image processing apparatus based on metadata added to the image received from the second another image processing apparatus.

17. The image processing system according to claim 16 , wherein

the metadata includes data indicating that an error is included in the image to which the metadata is added.

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 24, 2019

From: YUSHIYA, AKIHIKO; OTA, YUYA

To: CANON KABUSHIKI KAISHA

Reel/Frame 048976/0148 →

Priority Claims (1)

JP JP2016-200472 · Oct 11, 2016 · national

Continuity (2)

Continuation PCTJP2017030697 · Aug 28, 2017

Related Publication 20190228565A1 · Jul 25, 2019