Information processing apparatus for warning user of collision with physical object in cross reality (XR) experience, information processing method, and storage medium
An information processing apparatus includes at least one processor, and a memory coupled to the at least one processor and storing instructions that, when executed by the processor, cause the processor to function as a first acquisition unit configured to acquire information about a position and an orientation of a display apparatus, a second acquisition unit configured to acquire information about a position of a physical object, and a warning unit configured to change a warning method to warn a user depending on whether the physical object is present in a line-of-sight direction of the user.
1 . A head mounted display apparatus worn by a user comprising:
at least one sensor that senses objects in a predetermined region surrounding the user;
a display device that displays images visible to eyes of the user;
at least one memory storing instructions; and
at least one processor that, upon execution of the instructions, causes the processor:
acquire information about a position and an orientation of the head mounted display apparatus,
acquire, from the at least one sensor, information about a position of a physical object in real-space and within the predetermined region,
divide the predetermined region surrounding the user into first region corresponding to a line of sight of the user wearing the head mounted display and a second region that is not in the line of sight of the user;
determine, based on the position information of the physical object acquired from the at least one sensor, whether the physical object is located in the first region or the second region;
set a warning method to warn a user regarding the physical object depending on whether the physical object is determined to be present in the first region or the second region;
display, on the display device, a mixed-reality image including a virtual environment and including a warning object generated in accordance with the set warning method, wherein
when the physical object is determined to be in the second region, the warning object is superimposed in the mixed-reality image at or near a center of a display area of the display device and with a size larger than a size of the warning object in the case where the physical object is determined to be in the first region, and
when the physical object is determined to be in the first region, the warning object is superimposed in the mixed-reality image in an area corresponding to the physical object.
2 . The head mounted display apparatus according to claim 1 , wherein execution of the instructions further causes the processor to display the warning object to warn the user in a case where the physical object is present in a region within a predetermined distance from the head mounted display apparatus.
3 . The head mounted display apparatus according to claim 2 ,
wherein execution of the stored instructions further causes the at least one processor to:
divide the region within the predetermined distance into a plurality of warning regions based on the information about the position and the orientation of the head mounted display apparatus, and
determine whether the physical object is present in each of the plurality of warning regions, and
wherein the warning method is changed to warn the user depending on which warning region of the plurality of warning regions the physical object is present.
4 . The head mounted display apparatus according to claim 3 , wherein execution of the instructions further causes the processor to divide the region within the predetermined distance into the plurality of warning regions based on information about a view frustum of the head mounted display apparatus.
5 . The head mounted display apparatus according to claim 1 , wherein execution of the instructions further causes the processor to display the warning object to warn the user, in a case where the physical object is not present in the line-of-sight direction of the user, using a warning method easier to be recognized by the user than a warning method used in a case where the physical object is present in the line-of-sight direction of the user.
6 . The head mounted display apparatus according to claim 5 , wherein execution of the instructions further causes the processor to display a first mark on the display device in the case where the physical object is not present in the line-of-sight direction, and displays a second mark smaller than the first mark on the head mounted display apparatus in the case where the physical object is present in the line-of-sight direction.
7 . The head mounted display apparatus according to claim 5 , wherein execution of the instructions further causes the processor to warn the user, in a case where the virtual object is present between the head mounted display apparatus and the physical object, using a warning method easier to be recognized by the user than a warning method used in a case where the virtual object is not present between the head mounted display apparatus and the physical object.
8 . The head mounted display apparatus according to claim 7 , wherein execution of the instructions further causes the processor to display a first mark on the display device of the head mounted display apparatus in the case where the virtual object is present between the head mounted display apparatus and the physical object, and displays a second mark smaller than the first mark on the display device of the head mounted display apparatus in the case where the virtual object is not present between the head mounted display apparatus and the physical object.
9 . The head mounted display apparatus according to claim 1 , wherein execution of the instructions further causes the processor to change the warning method to warn the user when a virtual object is present between the head mounted display apparatus device and the physical object and in a case where the physical object is present in the line-of-sight direction of the user.
10 . The head mounted display apparatus according to claim 1 , wherein execution of the instructions further causes the processor to warn the user using at least one of the display device on the head mounted display apparatus, a sound, and a vibration.
11 . The head mounted display apparatus according to claim 1 , wherein execution of the instructions further causes the processor to not warn the user in a case where the physical object is present in the line-of-sight direction of the user, and warn the user in a case where the physical object is not present in the line-of-sight direction of the user.
12 . The head mounted display apparatus according to claim 1 , wherein execution of the instructions further causes the processor to acquire information about an orientation of the physical object.
13 . The head mounted display apparatus according to claim 1 , wherein execution of the instructions further causes the processor to display acquire information about a shape of the physical object.
14 . The head mounted display apparatus according to claim 1 , wherein the first region is determined based on a view frustum of the display device determined based on the position and the orientation of the head mounted display apparatus and angle-of-view information of the display device.
15 . The head mounted display apparatus according to claim 14 , wherein the first region comprises an overlapped region of a left-eye view frustum and a right-eye view frustum.
16 . The head mounted display apparatus according to claim 1 , wherein, when a virtual object is present between the head mounted display apparatus and the physical object and the physical object is determined to be in the first region, the warning method includes causing the virtual object to be displayed semi-transparently and superimposing the warning object in an area corresponding to the physical object.
17 . The head mounted display apparatus according to claim 1 , wherein, when an amount of change of the position and the orientation of the head mounted display apparatus is less than or equal to a threshold value for at least a predetermined time, the first region is set to a concentration region.
18 . The head mounted display apparatus according to claim 17 , wherein the concentration region is within an angle of view of 30 degrees in both vertical and horizontal directions from a center of an overlapped region of view frustums of the user's left eye and right eye.
19 . An information processing method executing on a head mounted display apparatus worn by a user and which includes at least one sensor that senses objects in a predetermined region surrounding the user and a display device that displays images visible to eyes of the user, the method comprising:
acquiring information about a position and an orientation of the head mounted display apparatus;
acquiring, from the at least one sensor, information about a position of a physical object in real space and within the predetermined region;
dividing the predetermined region surrounding the user into first region corresponding to a line of sight of the user wearing the head mounted display and a second region that is not in the line of sight of the user;
determining, based on the position information of the physical object acquired from the at least one sensor, whether the physical object is located in the first region or the second region;
setting a warning method to warn a user regarding the physical object depending on whether the physical object is determined to be present in the first region or the second region;
displaying, on the display device, a mixed-reality image including a virtual environment and including a warning object generated in accordance with the set warning method, wherein
when the physical object is determined to be in the second region, the warning object is superimposed in the mixed-reality image at or near a center of a display area of the display device and with a size larger than a size of the warning object in the case where the physical object is determined to be in the first region, and
when the physical object is determined to be in the first region, the warning object is superimposed in the mixed-reality image in an area corresponding to the physical object.
20 . A non-transitory computer-readable storage medium storing a program that causes a computer of a head mounted display apparatus worn by a user and which includes at least one sensor that senses objects in a predetermined region surrounding the user and a display device that displays images visible to eyes of the user to execute an information processing method, the information processing method comprising:
acquiring information about a position and an orientation of a-the head mounted display apparatus;
acquiring, from the at least one sensor, information about a position of a physical object in real space and within the predetermined region;
dividing the predetermined region surrounding the user into first region corresponding to a line of sight of the user wearing the head mounted display and a second region that is not in the line of sight of the user;
determining, based on the position information of the physical object acquired from the at least one sensor, whether the physical object is located in the first region or the second region;
setting a warning method to warn a user regarding the physical object depending on whether the physical object is determined to be present in the first region or the second region;
displaying, on the display device, a mixed-reality image including a virtual environment and including a warning object generated in accordance with the set warning method, wherein
when the physical object is determined to be in the second region, the warning object is superimposed in the mixed-reality image at or near a center of a display area of the display device and with a size larger than a size of the warning object in the case where the physical object is determined to be in the first region, and
when the physical object is determined to be in the first region, the warning object is superimposed in the mixed-reality image in an area corresponding to the physical object.