IP Library Granted Patent US 9,639,989
Granted Patent B2
US 9,639,989 · App. 14/406,919 · Granted May 2, 2017

Video processing device, video processing method, and video processing system

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,639,989
App. No.
14/406,919
Granted
May 2, 2017
Kind
B2
Abstract

An AR image generator 526 employs, as a marker, a subject imaged by an imaging element that images the subject existing in a region including the field of view of a user who wears an optically transmissive HMD and is provided for the optically transmissive HMD, and generates an AR image to be presented by the optically transmissive HMD. A proximity acquirer 560 acquires the distance in a virtual three-dimensional space between a first AR image associated with a first marker and a second marker or a second AR image associated with the second marker. An interaction producer 570 calculates an interaction caused with the first AR image or the second AR image based on the distance acquired by the proximity acquirer 560 . The AR image generator 526 changes at least either one image of the first AR image or the second AR image according to the interaction calculated by the interaction producer 570.

Claims (48)

1. A video processing device comprising:

an AR image generator that employs, as a marker, a subject imaged by an imaging element provided for an optically transmissive HMD and generates an AR (Augmented Reality) image to be presented by the optically transmissive HMD, the optically transmissive HMD presenting video observed when a three-dimensional image in a virtual three-dimensional space is projected into a real space, the imaging element imaging the subject existing in a region including a field of view of a user who wears the optically transmissive HMD;

a proximity acquirer that acquires proximity in the virtual three-dimensional space between a first AR image associated with a first marker and a second marker or a second AR image associated with the second marker,

wherein the proximity is calculated by comparing a sphere having a first radius centered at the first marker with a sphere having a second radius centered at the second marker to determine if the first sphere and the second sphere intersect; and

an interaction producer that calculates an interaction caused with the first AR image or the second AR image based on the intersection determined by the proximity acquirer;

the AR image generator changing at least either one image of the first AR image or the second AR image according to the interaction calculated by the interaction producer,

wherein the first AR image is composed of a plurality of polygons, and

wherein a position coordinate of the first marker is adaptively set at a coordinate of a polygon of the plurality of polygons having a shortest distance to the second marker.

2. The video processing device according to claim 1 , further comprising

an attribute managing section that manages an attribute including a virtual physical property given to each of AR images generated by the AR image generator;

the AR image generator changing an image to be generated when the attribute changes.

3. The video processing device according to claim 2 , wherein the attribute managing section changes the attribute given to the first AR image or the second AR image according to the interaction calculated by the interaction producer.

4. The video processing device according to claim 2 , wherein

the physical property included in the attribute managed by the attribute managing section includes at least one of position coordinates, velocity, acceleration, or mass of the AR image in the virtual three-dimensional space, and

the interaction producer changes the physical property of the first AR image and the physical property of the second AR image based on the intersection determined by the proximity acquirer.

5. The video processing device according to claim 2 , wherein the attribute managed by the attribute managing section includes information indicating a visual effect to be reflected in the AR image.

6. A video processing method comprising:

generating an AR image that is to be presented by a stereoscopic video observation device including an optically transmissive HMD and an imaging element and employs a subject imaged by the imaging element as a marker, the optically transmissive HMD presenting video observed when a three-dimensional image in a virtual three-dimensional space is projected into a real space, the imaging element imaging the subject existing in a region including a field of view of a user who wears the optically transmissive HMD;

acquiring proximity in the virtual three-dimensional space between a first AR image associated with a first marker and a second marker or a second AR image associated with the second marker,

wherein the proximity is calculated by comparing a sphere having a first radius centered at the first marker with a sphere having a second radius centered at the second marker to determine if the first sphere and the second sphere intersect;

calculating an interaction caused with the first AR image or the second AR image based on the determined intersection; and

changing at least either one image of the first AR image or the second AR image according to the calculated interaction,

wherein the first AR image is composed of a plurality of polygons, and

wherein a position coordinate of the first marker is adaptively set at a coordinate of a polygon of the plurality of polygons having a shortest distance to the second marker.

7. A non-transitory computer-readable medium having executable programming instructions stored thereon, the program comprising:

generating an AR image that is to be presented by a stereoscopic video observation device including an optically transmissive HMD and an imaging element and employs a subject imaged by the imaging element as a marker, the optically transmissive HMD presenting video observed when a three-dimensional image in a virtual three-dimensional space is projected into a real space, the imaging element imaging the subject existing in a region including a field of view of a user who wears the optically transmissive HMD;

acquiring proximity in the virtual three-dimensional space between a first AR image associated with a first marker and a second marker or a second AR image associated with the second marker,

wherein the proximity is calculated by comparing a sphere having a first radius centered at the first marker with a sphere having a second radius centered at the second marker to determine if the first sphere and the second sphere intersect;

calculating an interaction caused with the first AR image or the second AR image based on the determined intersection; and

changing at least either one image of the first AR image or the second AR image according to the calculated interaction,

wherein the first AR image is composed of a plurality of polygons, and

wherein a position coordinate of the first marker is adaptively set at a coordinate of a polygon of the plurality of polygons having a shortest distance to the second marker.

8. A video processing system comprising:

an optically transmissive HMD that presents video observed when a three-dimensional image in a virtual three-dimensional space is projected into a real space;

an imaging element that is provided for the optically transmissive HMD and images a subject existing in a region including a field of view of a user who wears the optically transmissive HMD;

an AR image generator that employs the subject imaged by the imaging element as a marker and generates an AR (Augmented Reality) image to be presented by the optically transmissive HMD;

a proximity acquirer that acquires proximity in the virtual three-dimensional space between a first AR image associated with a first marker and a second marker or a second AR image associated with the second marker,

wherein the proximity is calculated by comparing a sphere having a first radius centered at the first marker with a sphere having a second radius centered at the second marker to determine if the first sphere and the second sphere intersect; and

an interaction producer that calculates an interaction caused with the first AR image or the second AR image based on the determined intersection by the proximity acquirer;

the AR image generator changing at least either one image of the first AR image or the second AR image according to the interaction calculated by the interaction producer,

wherein the first AR image is composed of a plurality of polygons, and

wherein a position coordinate of the first marker is adaptively set at a coordinate of a polygon of the plurality of polygons having a shortest distance to the second marker.

9. The video processing device according to claim 1 , wherein the first radius is directly proportional to a size of the first AR image and the second radius is directly proportional to a size of the second AR image.

10. The video processing device according to claim 1 , wherein the first radius is a distance measured from the first marker to a farthest point on the first AR image.

11. The video processing device according to claim 1 ,

the first radius is set as an average of distances from the first marker to each of the plurality of polygons of the first AR image.

12. The video processing device according to claim 1 ,

the first radius is set as a minimum of distances from the first marker to each of the plurality of polygons of the first AR image.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 6, 2017
From: SONY CORPORATION
To: SONY INTERACTIVE ENTERTAINMENT INC.
Reel/Frame 043761/0975 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 25, 2017
From: SONY INTERACTIVE ENTERTAINMENT INC.
To: SONY CORPORATION
Reel/Frame 041083/0545 →
CHANGE OF NAME Recorded Jan 19, 2017
From: SONY COMPUTER ENTERTAINMENT INC.
To: SONY INTERACTIVE ENTERTAINMENT INC.
Reel/Frame 041015/0010 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 10, 2014
From: OHASHI, YOSHINORI
To: SONY COMPUTER ENTERTAINMENT INC.
Reel/Frame 034463/0840 →