Terminal device, service, and method for generating novel synthetic views using neural network-based volumn rendering
A terminal device ( 1 ) comprising circuitry configured to obtain a novel synthetic view based on a position and orientation of the terminal device ( 1 ) and to display the novel synthetic view to a user of the terminal device ( 1 ).
1 . A terminal device comprising:
circuitry configured to
generate, by a neural network that outputs a color value and a volume density based on a viewing position and a beam direction, a novel synthetic view based on a position and orientation of the terminal device; and
display the novel synthetic view to a user of the terminal device.
2 . The terminal device of claim 1 wherein the circuitry is configured to generate the novel synthetic view based on the position and orientation of the terminal device and based on imaging characteristics of the terminal device.
3 . The terminal device of claim 1 wherein the novel synthetic view generator represents an object, a specific location, a room, or a shop.
4 . The terminal device of claim 1 wherein the novel synthetic view allows a user of the terminal device to see through walls.
5 . The terminal device of claim 1 wherein the novel synthetic view allows a user of the look into a product packaging.
6 . The terminal device of claim 1 , wherein the circuitry is configured to recognize an obstructed area within an image obtained by a camera of the terminal device and wherein the novel synthetic view is representative of the obstruction area.
7 . The terminal device of claim 1 wherein the novel synthetic view is an updated synthetic view.
8 . The terminal device of claim 1 wherein the circuitry is configured to generate the novel synthetic view based on imaging characteristics of the terminal device.
9 . The terminal device of claim 8 wherein the imaging characteristics comprise at least one of focal length, display size, and number of pixels.
10 . The terminal device of claim 1 wherein the terminal device is a smartglasses device.
11 . The terminal device of claim 1 , wherein the circuitry is configured to generate, based on the position and orientation of the terminal device, a novel synthetic view for a left eye display and a novel synthetic view for a right eye display.
12 . The terminal device of claim 1 , wherein the novel synthetic view is associated with a layer selected from a group of predefined layers.
13 . The terminal device of claim 12 , wherein each layer is associated with at least one neural network configured to generate the novel synthetic view.
14 . The terminal device of claim 12 wherein the circuitry is configured to select the layer based on the position and orientation of the terminal device and/or based on information obtained by eye tracking.
15 . The terminal device of claim 12 wherein the circuitry is configured to provide selection means to the user, the selection means comprising a virtual menu presented to the user or means that analyses voice input of the user.
16 . The terminal device according to claim 1 , wherein the circuitry is configured to:
generate, for each pixel of a display of the terminal device, a virtual camera ray based on the position and orientation of the terminal device;
sample a plurality of points along the virtual camera ray, each point defining a 5D coordinate comprising a spatial location and a viewing direction;
input the 5D coordinates into the neural network to generate the color value and the volume density for each of the plurality of points; and
apply volume rendering to project the color values and volume densities of the plurality of points into the novel synthetic view.
17 . A service comprising:
circuitry configured to
generate, by a neural network that outputs a color value and a volume density based on a viewing position and a beam direction, a novel synthetic view based on a position and orientation of a terminal device; and
display the novel synthetic view to the user via a display of the terminal device.
18 . The service of claim 17 wherein the circuitry implements the neural network, the neural network configured to generate the novel synthetic view based on the position and orientation of the terminal device.
19 . The service of claim 17 wherein the neural network represents an object or room.
20 . A method comprising:
generating by a neural network that outputs a color value and a volume density based on a viewing position and a beam direction, a novel synthetic view based on a position and orientation of a terminal device; and
displaying the novel synthetic view to a user of the terminal device.