AR display apparatus and AR display method

1. An Augmented Reality (AR) display apparatus to display an AR image, being image information on augmented reality, onto a screen transmitting an actual view of an external world, the AR display apparatus comprising:

a processor, which when executing a program, configures the processor to:

determine whether an object in an image photographed by a camera is a fixed object or a moving object, the fixed object being an object that does not move, the camera being configured to photograph an area including an actual view corresponding to a field of view of a user, the moving object being an object that is moving;

extract, in a case where the processor determines that the object is the fixed object, a pixel in an area of the fixed object with respect to a plurality of image frames;

obtain an amount of change in a position of the pixel extracted with respect to the plurality of image frames;

judge visibility of the user to the fixed object in the actual view of the external world viewed through the screen on a basis of the amount of change in the position of the pixel;

control the AR image regarding the fixed object;

calculate a moving direction of the fixed object on a basis of a difference between the area of the fixed object at a first point in time and the area of the fixed object a second point in time, the area of the fixed object changing in time series on the screen transmitting the actual view of the external world;

predict the area of the fixed object at a third point in time on a basis of the area of the fixed object at the second point in time and the calculated moving direction; and

display the AR image of the fixed object at a position different from the predicted area of the fixed object at the third point in time,

wherein the different position where the AR image is displayed is a position that does not overlap with a predicted movement trajectory of the area of the fixed object based on the moving direction,

wherein the different position where the AR image is displayed is temporally ahead in the moving direction,

wherein different positions that are positions where the AR image is displayed are selected as neighboring positions when following the area of the fixed object at the third point in time,

wherein the AR image is displayed with a selected position as a static position for a period of time including at least the third point in time, and

wherein the different position where the AR image is displayed is a position above the area of the fixed object at the third point in time when the predicted movement trajectory is a horizontal movement trajectory.

2. The AR display apparatus according to claim 1 ,

wherein the object is classified into at least two classifications including first classification and second classification on the basis of the visibility,

wherein with respect to the object of the first classification, the processor is further configured to control the object so that the AR image is not displayed, or so that the AR image is generated and displayed without performing the image processing for the object, and

wherein with respect to the object of the second classification, the processor is further configured to control the object so that the object is subjected to the image processing to generate and display the AR image.

3. The AR display apparatus according to claim 1 ,

wherein the object is classified into at least three classifications including first classification, second classification, and third classification on the basis of the visibility,

wherein with respect to the object of the first classification, the processor is further configured to control the object so that the AR image is generated and displayed without performing the image processing for the object,

wherein with respect to the object of the second classification, the processor is further configured to control the object so that the object is subjected to first processing to generate and display the AR image, and

wherein with respect to the object of the third classification, the processor is further configured to control the object so that the object is subjected to second processing to generate and display the AR image.

4. The AR display apparatus according to claim 1 ,

wherein a central area is set in the screen, and

wherein the AR image is displayed at a position outside the central area in the screen.

5. The AR display apparatus according to claim 1 ,

wherein the image processing is an enlarging process to enlarge an image of the area of the object.

6. The AR display apparatus according to claim 1 ,

wherein the image processing contains optical character recognition (OCR) processing to generate a character image by using character information extracted from an image of the object.

7. The AR display apparatus according to claim 1 ,

wherein the image processing contains an attitude changing process to convert attitude of an image of the area of the object so as to become a state of a front face when viewed from a point of view of the user.

8. The AR display apparatus according to claim 1 , wherein the processor is further configured to:

track a line of sight of the user, and set a sight line central area on a basis of a point of regard, the point of regard being an intersection between a front of the line of sight and the screen; and

extract the object from the sight line central area.

9. The AR display apparatus according to claim 1 , wherein the processor is further configured to:

execute retrieval in a database, basic information and related information regarding the object being registered in the database,

wherein the processor is further configured to execute retrieval by being configured to: retrieve the basic information or related information from the database on the basis of information on the object extracted from the image; and generate the AR image by using retrieval result information that is a retrieval result, the AR image including the basic information or the related information regarding the object.

10. The AR display apparatus according to claim 9 ,

wherein an object image of the object is also registered in the database, and

wherein the processor is further configured to execute retrieval by being configured to: retrieve the database as an image on the basis of an image of the object extracted from the image; and use the object image as a retrieval result to generate the AR image.

11. The AR display apparatus according to claim 1 ,

wherein the AR display apparatus is a head up display apparatus mounted in an on-vehicle system, and

wherein the AR display apparatus further comprises:

a display controller configured to execute projection display onto the screen, the screen being configured by a front shield of a vehicle.

12. The AR display apparatus according to claim 1 ,

wherein the fixed object includes a signboard.

13. The AR display apparatus according to claim 1 ,

wherein the AR image is controlled to be displayed in a format of attaching a line that connects the object to the AR image.

14. The AR display apparatus according to claim 1 ,

wherein the static position is located in a vicinity of a center of an upper side area in the screen.

15. The AR display apparatus according to claim 1 ,

wherein a display position of the AR image regarding the fixed object is controlled so as to be set to a neighboring position of a moving range in which the area of the fixed object changes in the time series.

16. The AR display apparatus according to claim 1 ,

wherein movement of the AR image in the screen is controlled so as to become slower than a speed of movement of the object.

17. The AR display apparatus according to claim 1 ,

wherein, in a case where a predetermined time elapses since the AR image is displayed, the AR image is controlled so as to be erased.

18. An Augmented Reality (AR) display method for an AR display apparatus to display an AR image, being image information on augmented reality, onto a screen transmitting an actual view of an external world, the AR display method comprising the steps of:

determining whether an object in an image photographed by a camera is a fixed object or a moving object, the fixed object being an object that does not move, the camera being configured to photograph an area including an actual view corresponding to a field of view of a user, the moving object being an object that is moving;

extracting, in a case where it is determined that the object is the fixed object, a pixel in an area of the fixed object with respect to a plurality of image frames;

obtaining an amount of change in a position of the pixel extracted with respect to the plurality of image frames;

judging visibility of the user to the fixed object in the actual view of the external world viewed through the screen on a basis of the amount of change in the position of the pixel;

controlling the AR image regarding the fixed object;

calculating a moving direction of the fixed object on a basis of a difference between the area of the fixed object at a first point in time and the area of the fixed object a second point in time, the area of the fixed object changing in time series on the screen transmitting the actual view of the external world;

predicting the area of the fixed object at a third point in time on a basis of the area of the fixed object at the second point in time and the calculated moving direction; and

displaying the AR image of the fixed object at a position different from the predicted area of the fixed object at the third point in time,

wherein the different position where the AR image is displayed is a position that does not overlap with a predicted movement trajectory of the area of the fixed object based on the moving direction,

wherein the different position where the AR image is displayed is temporally ahead in the moving direction,

wherein different positions that are positions where the AR image is displayed are selected as neighboring positions when following the area of the fixed object at the third point in time,

wherein the AR image is displayed with a selected position as a static position for a period of time including at least the third point in time, and

wherein the different position where the AR image is displayed is a position above the area of the fixed object at the third point in time when the predicted movement trajectory is a horizontal movement trajectory.