Granted Patent B2

US 10,129,524 · App. 15/439,710 · Granted Nov 13, 2018

Inventors: Yi-Ren Ng (Palo Alto, CA); Eric Cheng (San Francisco, CA); Chia-Kai Liang (San Jose, CA); Kayvon Fatahalian (Pittsburgh, PA); David John Evans (Palo Alto, CA); Kira Wampler (Mountain View, CA); Kristen Berman (San Francisco, CA); Kurt Barton Akeley (Saratoga, CA)

Assignee: GOOGLE LLC

H04N13/128G06F3/013G06F21/31

Abstract

Claims (223)

1. A method for manipulating depth-assigned content associated with a depth-enhanced virtual reality image, comprising:

at a processor, receiving a depth-enhanced virtual reality image comprising a plurality of elements having different depth values;

at the processor, receiving a depth-assigned content item, the depth-assigned content item being associated with the depth-enhanced virtual reality image and being assigned to a first depth value;

at the processor, determining a focus depth for rendering of the depth-enhanced virtual reality image, wherein at least a first one of the elements has a depth that is closer to the focus depth than the depth of at least a second one of the elements;

at the processor, determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image based on a determination as to whether the first depth value falls within a predetermined tolerance value of the determined focus depth;

at an image rendering component, rendering the depth-enhanced virtual reality image so that the first element is more in focus than the second element;

at the image rendering component:

responsive to the determination indicating that the depth-assigned content item should be included, including the depth-assigned content item in the rendered depth-enhanced virtual reality image;

responsive to the determination indicating that the depth-assigned content item should not be included, omitting the depth-assigned content item from the rendered depth-enhanced virtual reality image; and

outputting the rendered depth-enhanced virtual reality image at an output device.

2. The method of claim 1 , wherein the depth-enhanced virtual reality image comprises a computer-generated image.

3. The method of claim 1 , wherein the depth-enhanced virtual reality image comprises a combination of a captured image and a computer-generated image.

4. The method of claim 1 , wherein:

the depth-assigned content item is assigned to a location within the depth-enhanced virtual reality image; and

rendering the depth-enhanced virtual reality image to generate a rendered image including the depth-assigned content item comprises rendering the depth-assigned content item at the location to which it is assigned.

5. The method of claim 4 , wherein the position within the depth-enhanced virtual reality image is indicated as a coordinate value in three-dimensional space.

6. The method of claim 1 , wherein determining a focus depth comprises receiving, at an input device, user input specifying a focus depth.

7. The method of claim 1 , wherein determining a focus depth comprises automatically determining the focus depth.

8. The method of claim 1 , wherein determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises determining whether the first depth value equals the determined focus depth.

9. The method of claim 1 , wherein rendering the depth-enhanced virtual reality image to generate a rendered image including the depth-assigned content item comprises:

responsive to the first depth value falling outside the predetermined tolerance value of the determined focus depth, rendering the depth-enhanced virtual reality image to generate a rendered image including a subdued representation of the depth-assigned content item.

10. The method of claim 9 , wherein the subdued representation of the depth-assigned content item comprises at least one selected from the group consisting of:

a grayed out representation of the depth-assigned content item; and

a blurred representation of the depth-assigned content item.

11. The method of claim 1 , further comprising:

at the processor, determining a second focus depth for rendering of the depth-enhanced virtual reality image;

at the processor, determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image, based on a determination as to whether the first depth value falls within a predetermined tolerance value of the determined second focus depth;

at the image rendering component, responsive to the determination indicating that the depth-assigned content item should be included, and responsive to the previous determination indicating that the depth-assigned content item should not be included, re-rendering the depth-enhanced virtual reality image to introduce the depth-assigned content item into the rendered image;

at the image rendering component, responsive to the determination indicating that the depth-assigned content item should not be included, and responsive to the previous determination indicating that the depth-assigned content item should be included, re-rendering the depth-enhanced virtual reality image to dismiss the depth-assigned content item from the rendered image;

outputting the rendered image at an output device.

12. The method of claim 1 , wherein the depth-assigned content item comprises at least one selected from the group consisting of:

a tag;

an annotation;

a comment;

a text item;

an image;

an interactive element;

a hyperlink;

a reference to a document; and

a reference to a resource.

13. The method of claim 1 , wherein the depth-assigned content item comprises a machine-readable code.

14. The method of claim 1 , wherein the depth-assigned content item comprises at least one selected from the group consisting of:

in-image content; and

off-image content.

15. The method of claim 1 , further comprising:

at a user input device, receiving user input indicating an interaction with the depth-enhanced virtual reality image; and

responsive to the interaction with the depth-enhanced virtual reality image, causing a change to the depth-assigned content item.

16. The method of claim 15 , wherein causing a change to the depth-assigned content item comprises causing at least one additional content item to be displayed.

17. The method of claim 15 , wherein receiving user input indicating an interaction with the depth-enhanced virtual reality image comprises collecting information about user interaction with the depth-enhanced virtual reality image, the method further comprising:

determining at least one point of interest based on the collected information.

18. The method of claim 15 , wherein receiving user input indicating an interaction with the depth-enhanced virtual reality image comprises determined a direction of user gaze, the method further comprising:

determining at least one point of interest based on the determined direction of user gaze.

19. The method of claim 1 , further comprising:

at a user input device, receiving user input indicating an interaction with the depth-assigned content item; and

responsive to the interaction with the depth-enhanced virtual reality image, causing a change to the depth-enhanced virtual reality image.

20. The method of claim 19 , wherein causing a change to the depth-enhanced virtual reality image comprises re-rendering the depth-enhanced virtual reality image at a depth responsive to the received user input.

21. The method of claim 20 , wherein re-rendering the depth-enhanced virtual reality image at a depth responsive to the received user input comprises bringing into focus an area indicated to be of interest based on the received user input.

22. The method of claim 20 , wherein re-rendering the depth-enhanced virtual reality image at a depth responsive to the received user input comprises emphasizing an area indicated to be of interest based on the received user input.

23. The method of claim 20 , wherein receiving user input indicating an interaction with the depth-assigned content item comprises receiving user input indicating an interaction with extrinsic content associated with the depth-assigned content item.

24. The method of claim 1 , wherein the depth-assigned content item is associated with an additional content item, the method further comprising:

at a user input device, receiving user input indicating an interaction with the depth-assigned content item; and

responsive to the interaction with the depth-enhanced virtual reality image, causing a change to the additional content item.

25. The method of claim 24 , wherein the additional content item comprises content separate from the image.

26. The method of claim 24 , wherein receiving user input indicating an interaction with the depth-enhanced content item comprises collecting information about user interaction with the depth-enhanced content item, the method further comprising:

determining at least one point of interest based on the collected information.

27. The method of claim 1 , further comprising:

receiving an indication of popularity of a depth-assigned content item in a social networking environment; and

responsive to the received indication, causing a change to the depth-enhanced virtual reality image.

28. The method of claim 1 , wherein the depth-assigned content, when included, selectively obscures an element of the virtual reality image, and wherein determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises determining whether a user is not authorized to view the element.

29. The method of claim 1 , wherein determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises determining whether a user is authorized to view the depth-assigned content.

30. The method of claim 29 , wherein determining whether a user is authorized to view the depth-assigned content comprises receiving user login information and performing user authentication.

31. The method of claim 1 , further comprising:

at the processor, receiving a predefined sequence of display depths; and

at the image rendering component, successively presenting the depth-enhanced virtual reality image at each display depth in the received sequence.

32. The method of claim 31 , wherein the predefined sequence of display depths specifies a time period for each of at least a subset of the display depths;

and wherein successively presenting the depth-enhanced virtual reality image at each display depth comprises, for each display depth, presenting the depth-enhanced virtual reality image for the specified time period.

33. The method of claim 31 , wherein successively presenting the depth-enhanced virtual reality image at each display depth in the received sequence comprises:

at the image rendering component, presenting the depth-enhanced virtual reality image at the first display depth in the sequence;

at the processor, receiving an indication of a trigger event for proceeding to the next display depth in the sequence;

at the image rendering component, responsive to the trigger event, presenting the depth-enhanced virtual reality image at the next display depth in the sequence; and

repeating the steps of receiving an indication of a trigger event and presenting the depth-enhanced virtual reality image at the next display depth, until the depth-enhanced virtual reality image has been presented at the last display depth in the sequence.

34. The method of claim 33 , wherein the trigger event comprises user input received at an input device.

35. The method of claim 31 , wherein successively presenting the depth-enhanced virtual reality image at each display depth in the sequence comprises, for each display depth, performing the steps of:

at the processor, determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image, based on a determination as to whether the depth value of the depth-assigned content item falls within a predetermined tolerance value of the display depth;

at an image rendering component, responsive to the determination indicating that the depth-assigned content item should be included, rendering the depth-enhanced virtual reality image to generate a rendered image including the depth-assigned content item;

at the image rendering component, responsive to the determination indicating that the depth-assigned content item should not be included, rendering the depth-enhanced virtual reality image to generate a rendered image omitting the depth-assigned content item; and

outputting the rendered image at the output device.

36. The method of claim 1 , wherein determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises determining readiness of a user to view the depth-assigned content item.

37. The method of claim 36 , wherein determining readiness of a user to view the depth-assigned content item comprises determining direction of user gaze to determine which portion of a plurality of portions of a related content item is currently being viewed by a user.

38. The method of claim 36 , wherein determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises:

at the processor, determining at least one affinity between the user and at least one other entity; and

at the processor, determining whether to include the depth-assigned content item based on at least one reaction of the at least one other entity to the depth-assigned content item.

39. The method of claim 38 , wherein the at least one other entity comprises at least one entity associated with the user via a social network.

40. The method of claim 38 , wherein determining whether to include the depth-assigned content item based on at least one reaction of the at least one other entity to the depth-assigned content item comprises determining whether to include the depth-assigned content item based on aggregated reactions of a plurality of other entities to the depth-assigned content item.

41. The method of claim 36 , wherein determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises:

at the processor, determining at least one affinity between the user and at least one other entity; and

at the processor, determining whether to include the depth-assigned content item based on at least one reaction of the at least one other entity to at least one content item having affinity to the depth-assigned content item.

42. The method of claim 1 , further comprising, prior to receiving the depth-enhanced virtual reality image:

associating at least one content item with a depth-enhanced virtual reality image; and

assigning each content item to at least one depth value.

43. A method for presenting a series of renderings of a depth-enhanced virtual reality image, comprising:

at a processor, receiving a depth-enhanced virtual reality image comprising a plurality of elements having different depth values;

at the processor, receiving at least one depth-assigned content item, each depth-assigned content item being associated with the depth-enhanced virtual reality image and being assigned to a content item depth value;

at the processor, receiving a predefined sequence of display depths for rendering of the depth-enhanced virtual reality image, wherein, for each display depth in the sequence, at least a first one of the elements has a depth that is closer to the display depth than the depth of at least a second one of the elements; and

at an image rendering component, successively presenting the depth-enhanced virtual reality image at each display depth in the sequence;

wherein successively presenting the depth-enhanced virtual reality image at each display depth in the sequence comprises, for each display depth in the sequence:

at the processor, determining whether to include each depth-assigned content item in rendering the depth-enhanced virtual reality image, based on a determination as to whether the depth value for each depth-assigned content item falls within a predetermined tolerance value of the display depth;

at an image rendering component, rendering the depth-enhanced picture so that the first element is more in focus than the second element;

at the image rendering component:

responsive to the determination indicating that at least one depth-assigned content item should be included, including the at least one depth-assigned content item in the rendered depth-enhanced virtual reality image; and

responsive to the determination indicating that no depth-assigned content item should be included, omitting the at least one depth-assigned content item from the rendered depth-enhanced virtual reality image; and

outputting the rendered virtual reality image at an output device.

44. A computer program product for manipulating depth-assigned content associated with a depth-enhanced virtual reality image, comprising:

a non-transitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of:

receiving a depth-enhanced virtual reality image comprising a plurality of elements having different depth values;

receiving a depth-assigned content item, the depth-assigned content item being associated with the depth-enhanced virtual reality image and being assigned to a first depth value;

determining a focus depth for rendering of the depth-enhanced virtual reality image, wherein at least a first one of the elements has a depth that is closer to the focus depth than the depth of at least a second one of the elements;

determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image based on a determination as to whether the first depth value falls within a predetermined tolerance value of the determined focus depth;

rendering the depth-enhanced virtual reality image so that the first element is more in focus than the second element;

responsive to the determination indicating that the depth-assigned content item should be included, including the depth-assigned content item in the rendered depth-enhanced virtual reality image;

responsive to the determination indicating that the depth-assigned content item should not be included, omitting the depth-assigned content item from the rendered depth-enhanced virtual reality image; and

causing an output device to output the rendered depth-enhanced virtual reality image.

45. The computer program product of claim 44 , wherein the depth-enhanced virtual reality image comprises a computer-generated image.

46. The computer program product of claim 44 , wherein the depth-enhanced virtual reality image comprises a combination of a captured image and a computer-generated image.

47. The computer program product of claim 44 , wherein:

the depth-assigned content item is assigned to a location within the depth-enhanced virtual reality image; and

the computer program code configured to cause at least one processor to render the depth-enhanced virtual reality image to generate a rendered image including the depth-assigned content item comprises computer program code configured to cause at least one processor to render the depth-assigned content item at the location to which it is assigned.

48. The computer program product of claim 44 , wherein the computer program code configured to cause at least one processor to determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises computer program code configured to cause at least one processor to determine whether the first depth value equals the determined focus depth.

49. The computer program product of claim 44 , wherein the computer program code configured to cause at least one processor to render the depth-enhanced virtual reality image to generate a rendered image including the depth-assigned content item comprises:

the computer program code configured to cause at least one processor to, responsive to the first depth value falling outside the predetermined tolerance value of the determined focus depth, render the depth-enhanced virtual reality image to generate a rendered image including a subdued representation of the depth-assigned content item.

50. The computer program product of claim 44 , wherein the depth-assigned content item comprises at least one selected from the group consisting of:

a tag;

an annotation;

a comment;

a text item;

an image;

an interactive element;

a hyperlink;

a reference to a document;

a reference to a resource;

a machine-readable code; and

a Quick Response (QR) code.

51. The computer program product of claim 44 , further comprising computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of:

receiving, via a user input device, user input indicating an interaction with the depth-enhanced virtual reality image; and

responsive to the interaction with the depth-enhanced virtual reality image, causing a change to the depth-assigned content item.

52. The computer program product of claim 44 , further comprising computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of:

receiving, via a user input device, user input indicating an interaction with the depth-assigned content item; and

responsive to the interaction with the depth-enhanced virtual reality image, causing a change to the depth-enhanced virtual reality image.

53. The computer program product of claim 44 , wherein the depth-assigned content item is associated with an additional content item, the computer program code further comprising computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of:

receiving, via a user input device, user input indicating an interaction with the depth-assigned content item; and

responsive to the interaction with the depth-enhanced virtual reality image, causing a change to the additional content item.

54. The computer program product of claim 44 , further comprising computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of:

receiving an indication of popularity of a depth-assigned content item in a social networking environment; and

responsive to the received indication, causing a change to the depth-enhanced virtual reality image.

55. The computer program product of claim 44 , wherein the depth-assigned content, when included, selectively obscures an element of the virtual reality image, and wherein the computer program code configured to cause at least one processor to determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises computer program code configured to cause at least one processor to determine whether a user is not authorized to view the element.

56. The computer program product of claim 44 , wherein the computer program code configured to cause at least one processor to determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises computer program code configured to cause at least one processor to determine whether a user is authorized to view the depth-assigned content.

57. The computer program product of claim 44 , further comprising computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of:

receiving a predefined sequence of display depths; and

successively presenting the depth-enhanced virtual reality image at each display depth in the received sequence.

58. The computer program product of claim 44 , wherein the computer program code configured to cause at least one processor to determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises computer program code configured to cause at least one processor to determine readiness of a user to view the depth-assigned content item.

59. The computer program product of claim 58 , wherein the computer program code configured to cause at least one processor to determine readiness of a user to view the depth-assigned content item comprises computer program code configured to cause at least one processor to determine direction of user gaze to determine which portion of a plurality of portions of a related content item is currently being viewed by a user.

60. The computer program product of claim 58 , wherein the computer program code configured to cause at least one processor to determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises computer program code configured to cause at least one processor to perform the steps of:

determining at least one affinity between the user and at least one other entity; and

determining whether to include the depth-assigned content item based on at least one reaction of the at least one other entity to the depth-assigned content item.

61. The computer program product of claim 44 , further comprising computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of, prior to receiving the depth-enhanced virtual reality image:

associating at least one content item with a depth-enhanced virtual reality image; and

assigning each content item to at least one depth value.

62. A system for manipulating depth-assigned content associated with a depth-enhanced virtual reality image, comprising:

a processor, configured to:

receive a depth-enhanced virtual reality image comprising a plurality of elements having different depth values;

receive a depth-assigned content item, the depth-assigned content item being associated with the depth-enhanced virtual reality image and being assigned to a first depth value;

determine a focus depth for rendering of the depth-enhanced virtual reality image, wherein at least a first one of the elements has a depth that is closer to the focus depth than the depth of at least a second one of the elements; and

determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image based on a determination as to whether the first depth value falls within a predetermined tolerance value of the determined focus depth;

an image renderer, communicatively coupled to the processor, configured to:

render the depth-enhanced virtual reality image so that the first element is more in focus than the second element;

responsive to the determination indicating that the depth-assigned content item should be included, include the depth-assigned content item in the rendered depth-enhanced virtual reality image; and

responsive to the determination indicating that the depth-assigned content item should not be included, omit the depth-assigned content item from the rendered depth-enhanced virtual reality image; and

an output device, communicatively coupled to the image rendered, configured to output the rendered depth-enhanced virtual reality image.

63. The system of claim 62 , wherein the depth-enhanced virtual reality image comprises a computer-generated image.

64. The system of claim 62 , wherein the depth-enhanced virtual reality image comprises a combination of a captured image and a computer-generated image.

65. The system of claim 62 , wherein:

the depth-assigned content item is assigned to a location within the depth-enhanced virtual reality image; and

the image renderer is configured to render the depth-enhanced virtual reality image to generate a rendered image including the depth-assigned content item at the location to which it is assigned.

66. The system of claim 62 , wherein the processor is configured to determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image by determining whether the first depth value equals the determined focus depth.

67. The system of claim 62 , wherein the image renderer is configured to, responsive to the first depth value falling outside the predetermined tolerance value of the determined focus depth, render the depth-enhanced virtual reality image to generate a rendered image including a subdued representation of the depth-assigned content item.

68. The system of claim 62 , wherein the depth-assigned content item comprises at least one selected from the group consisting of:

a tag;

an annotation;

a comment;

a text item;

an image;

an interactive element;

a hyperlink;

a reference to a document;

a reference to a resource;

a machine-readable code; and

a Quick Response (QR) code.

69. The system of claim 62 , further comprising:

a user input device, configured to receive user input indicating an interaction with the depth-enhanced virtual reality image;

wherein the processor is configured to, responsive to the interaction with the depth-enhanced virtual reality image, cause a change to the depth-assigned content item.

70. The system of claim 62 , further comprising:

a user input device, configured to receive user input indicating an interaction with the depth-assigned content item;

wherein the processor is configured to, responsive to the interaction with the depth-enhanced virtual reality image, cause a change to the depth-enhanced virtual reality image.

71. The system of claim 62 , wherein the depth-assigned content item is associated with an additional content item, the system further comprising:

a user input device, configured to receive user input indicating an interaction with the depth-assigned content item;

wherein the processor is configured to, responsive to the interaction with the depth-enhanced virtual reality image, cause a change to the additional content item.

72. The system of claim 62 , wherein the processor is further configured to:

receive an indication of popularity of a depth-assigned content item in a social networking environment; and

responsive to the received indication, cause a change to the depth-enhanced virtual reality image.

73. The system of claim 62 , wherein the depth-assigned content, when included, selectively obscures an element of the virtual reality image, and wherein the processor is configured to determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image by determining whether a user is not authorized to view the element.

74. The system of claim 62 , wherein the processor is configured to determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image by determining whether a user is authorized to view the depth-assigned content.

75. The system of claim 62 , wherein:

the processor is configured to receive a predefined sequence of display depths; and

the output device is configured to successively present the depth-enhanced virtual reality image at each display depth in the received sequence.

76. The system of claim 62 , wherein the processor is configured to determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image by determining readiness of a user to view the depth-assigned content item.

77. The system of claim 76 , wherein the processor is configured to determine readiness of a user to view the depth-assigned content item by determining direction of user gaze to determine which portion of a plurality of portions of a related content item is currently being viewed by a user.

78. The system of claim 76 , wherein the processor is configured to determine whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image by:

determining at least one affinity between the user and at least one other entity; and

determining whether to include the depth-assigned content item based on at least one reaction of the at least one other entity to the depth-assigned content item.

79. The system of claim 62 , wherein the processor is configured to, prior to receiving the depth-enhanced virtual reality image:

associate at least one content item with a depth-enhanced virtual reality image; and

assign each content item to at least one depth value.

Assignments (3)

Continuity (2)

Cited By (20)

US 12,204,281 US 12,288,408 US 12,327,392 US 12,333,488 US 12,361,375 US 12,367,457 US 12,374,115 US 12,380,400 US 12,412,149 US 12,412,355 US 12,430,608 US 12,437,263 US 12,450,558 US 12,450,883 US 12,469,005 US 12,469,255 US 12,524,902 US 12,567,167 US 12,591,847 US 12,705,850