Computer-readable storage medium having information processing program stored therein, information processing apparatus, information processing system, and information processing method

1. A non-transitory computer-readable storage medium having stored therein an information processing program executable by a computer of an information processing apparatus, the program, when executed, causing the computer to perform operations comprising:

obtaining a taken image that is taken by a camera of an imaging apparatus;

obtaining orientation information from a sensor included in the imaging apparatus, the orientation information indicating an orientation of the camera;

detecting a specific object from the taken image;

calculating a relative positional relation between the specific object and the camera;

determining virtual object information including a virtual object display form, a virtual object type, and/or a number of virtual objects, based on the orientation information;

generating an image of one or more virtual objects corresponding to the virtual object information based on the calculated positional relation between the specific object and the camera; and

displaying the image of the one or more virtual objects on a display device.

2. The non-transitory computer-readable storage medium according to claim 1 , wherein the virtual object display form is determined on the basis of at least the orientation information.

3. The non-transitory computer-readable storage medium according to claim 1 , wherein the virtual object display form, the virtual object type, and/or the number of virtual objects, is determined on the basis of the positional relation and the orientation information.

4. The non-transitory computer-readable storage medium according to claim 3 , wherein the virtual object display form is determined in accordance with an orientation of the specific object with respect to a predetermined direction indicated by the orientation information.

5. The non-transitory computer-readable storage medium according to claim 4 , wherein

the obtained orientation information comprises orientation information indicating a predetermined direction in a coordinate system based on the camera, and

a predetermined direction in a coordinate system is calculated based on the specific object on the basis of the positional relation and the obtained orientation information, and the virtual object display form is determined on the basis of the calculated predetermined direction.

6. The non-transitory computer-readable storage medium according to claim 5 , wherein

at least when the calculated predetermined direction indicates that the specific object is located in a real space so as to be perpendicular to the predetermined direction indicated by the orientation information, the virtual object display form is determined as a first display form, and

at least when the calculated predetermined direction indicates that the specific object is located in the real space so as to be parallel to the predetermined direction indicated by the orientation information, the virtual object display form is determined as a second display form.

7. The non-transitory computer-readable storage medium according to claim 4 , wherein an orientation of the virtual object with respect to the specific object is determined in accordance with the orientation of the specific object with respect to the predetermined direction indicated by the orientation information.

8. The non-transitory computer-readable storage medium according to claim 7 , wherein

at least when the specific object is located in a real space so as to be perpendicular to the predetermined direction indicated by the orientation information, the orientation of the virtual object is determined such that the virtual object has an orientation perpendicular to the specific object, and

at least when the specific object is located in the real space so as to be parallel to the predetermined direction indicated by the orientation information, the orientation of the virtual object is determined such that the virtual object has an orientation parallel to the specific object.

9. The non-transitory computer-readable storage medium according to claim 8 , wherein at least when the specific object is located in the real space so as to be parallel to the predetermined direction indicated by the orientation information, the orientation of the virtual object is determined as a first orientation if a rotation angle of at least a surface of the specific object about an axis perpendicular to the surface is a first angle, and the orientation of the virtual object is determined as a second orientation if the rotation angle is a second angle.

10. The non-transitory computer-readable storage medium according to claim 4 , wherein the predetermined direction is a direction of gravity.

11. The non-transitory computer-readable storage medium according to claim 5 , wherein

when the predetermined direction in the coordinate system based on the specific object is in a first range, the virtual object display form is determined as a first form,

when the predetermined direction in the coordinate system based on the specific object is in a second range that is separated from the first range by a predetermined boundary, the virtual object display form is determined as a second form, and

when the display form of the virtual object is changed from the first form to the second form, the predetermined boundary is set on the first range side as compared to that when the virtual object display form is changed from the second form to the first form.

12. The non-transitory computer-readable storage medium according to claim 1 , wherein the image of the one or more virtual objects is displayed on the display device such that the image is superimposed on the taken image that is taken by the camera or a real space that is viewed through a screen of the display device.

13. The non-transitory computer-readable storage medium according to claim 9 , wherein

first information indicating the first orientation and second information indicating the second orientation are provided on the specific object,

the first information is visible from an appropriate direction when the specific object is viewed from a first direction, and

the second information is visible from an appropriate direction when the specific object is viewed from a second direction.

14. The non-transitory computer-readable storage medium according to claim 1 , wherein

the sensor is an orientation sensor configured to detect the orientation of the camera.

15. An information processing apparatus comprising:

a camera configured to capture a taken image;

a sensor configured to obtain orientation information indicating an orientation of the camera; and

at least one processor being configured to:

detect a specific object from the taken image captured by the camera;

calculate a relative positional relation between the specific object and the camera;

determine virtual object information including a virtual object display form, a virtual object type, and/or a number of virtual objects, based on the orientation information;

generate an image of one or more virtual objects corresponding to the virtual object information based on the calculated positional relation between the specific object and the camera; and

display the image of the one or more virtual objects on a display device.

16. An information processing system comprising:

a camera configured to capture a taken image;

a sensor configured to obtain orientation information indicating an orientation of the camera; and

at least one processor being configured to:

detect a specific object from the taken image obtained by the camera;

calculate a relative positional relation between the specific object and the camera;

determine virtual object information including a virtual object display form, a virtual object type, and/or a number of virtual objects, based on the orientation information;

generate an image of one or more virtual objects corresponding to the virtual object information based on the calculated positional relation between the specific object and the camera; and

display the image of the one or more virtual objects on a display device.

17. An information processing method comprising:

obtaining a taken image that is taken by a camera of an imaging apparatus;

obtaining orientation information from a sensor in the imaging apparatus, the orientation information indicating an orientation of the camera;

detecting a specific object from the taken image;

calculating a relative positional relation between the specific object and the camera;

determining virtual object information including a virtual object display form, a virtual object type, and/or a number of virtual objects, based on at least the orientation information;

generating an image of one or more virtual objects corresponding to the virtual object information based on the calculated positional relation; and

displaying the image of the one or more virtual objects on a display device.

18. An information processing system comprising an information processing apparatus and a marker,

the information processing apparatus comprising:

a camera;

a sensor configured to detect an orientation of the camera; and

at least one processor coupled to the camera and the sensor, the at least one processor being configured to:

take an image of the marker using the camera;

obtain the taken image;

obtain orientation information indicating an orientation of the camera from the sensor;

detect the marker from the taken image;

calculate a relative positional relation between the marker and the camera;

determine virtual object information including a virtual object display form, a virtual object type, and/or a number of virtual objects, based on at least the orientation information;

generate an image of one or more virtual objects corresponding to the virtual object information based on the calculated positional relation; and

display the image of the one or more virtual objects on a display device.

19. The non-transitory computer-readable storage medium according to claim 1 , wherein the sensor is an acceleration sensor.

20. The non-transitory computer-readable storage medium according to claim 1 , wherein the program, when executed, further causes the computer to rotate the displayed virtual object in accordance with a rotation of the camera about a central axis in the real space.

21. An information processing apparatus comprising:

a camera for capturing an image;

a gravity sensor commonly housed with the camera;

a display; and

processing circuitry coupled to the camera, the gravity sensor and the display, the processing circuitry detecting a marker from the image captured by the camera, calculating a relative positional relationship between the marker and the camera, determining virtual object information including a virtual object display form, a virtual object type, and/or a number of virtual objects, based on at least a direction of gravity determined in accordance with an output of the gravity sensor, generating one or more virtual objects based on the virtual object information and in accordance with the calculated positional relationship between the marker and the camera, and controlling the display to display the one or more virtual objects.

22. The information processing apparatus according to claim 21 , wherein the camera is a stereo camera and the image is a stereo image.

23. The information processing apparatus according to claim 21 , wherein the gravity sensor comprises an accelerometer.