Determining an assignment of virtual objects to positions in a user field of view to render in a mixed reality display
Provided are a computer program product, system, and method for determining an assignment of virtual objects to positions in a user field of view to render in a mixed reality display. A reward score for a virtual object is calculated based on an easiness score indicating an easiness to view the virtual object in previous interactions with the virtual object and frequency of interactions with the virtual object. The reward scores and the viewability scores for the positions are inputted into a virtual object placement program to output an assignment of the virtual objects to the positions. The assignment of the virtual objects to the positions are transmitted to the mixed reality display to cause the mixed reality display to render the virtual objects at the positions in the field of view of the user indicated in the assignment.
1 . A computer program product for determining virtual objects to render in a mixed reality display, the computer program product comprising a computer readable storage medium having computer readable program code embodied therein that is executable to perform operations, the operations comprising:
providing viewability scores for positions in a field of view of a user in which virtual objects are rendered, wherein a viewability score for a position is based on a distance from the user and change in viewing angle to view the position;
calculating reward scores for virtual objects, wherein a reward score for a virtual object is calculated based on an easiness score indicating an easiness to view the virtual object in previous interactions with the virtual object and frequency of interactions with the virtual object;
inputting the reward scores and the viewability scores for the positions into a virtual object placement program to output an assignment of the virtual objects to the positions; and
transmitting the assignment of the virtual objects to the positions to the mixed reality display to cause the mixed reality display to render the virtual objects at the positions in the field of view of the user indicated in the assignment.
2 . The computer program product of claim 1 , wherein the virtual object placement program implements machine learning, wherein the operations further comprise:
receiving feedback indicating interactions with the virtual objects rendered at the positions according to the assignment, wherein the virtual object placement program outputted the assignment with a confidence level indicating a probability that the assignment is optimal;
determining whether the interactions satisfy a positive threshold indicating a high level of interactions with the virtual objects in the positions indicated in the assignment or a low threshold indicating a low level of interactions with the virtual objects in the positions indicated in the assignment;
training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with an increased confidence level comprising an increase to the confidence level in response to determining that the interactions satisfy the positive threshold; and
training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with a decreased confidence level comprising a decrease to the confidence level in response to determining that the interactions do not satisfy the positive threshold.
3 . The computer program product of claim 1 , wherein the easiness to view a virtual object is a function of a sum of a weight based on a depth of the user from the virtual object and a change in a viewing angle by the user to view the virtual object in the field of view times a size of the virtual object in the field of view for previous interactions with the virtual object.
4 . The computer program product of claim 1 , wherein calculating a reward score for a virtual object further comprises:
calculating an interaction score based on user previous interactions with the virtual object, wherein the reward score is a function of the easiness score and the interaction score.
5 . The computer program product of claim 4 , wherein calculating the interaction score for the virtual object further comprises calculating an implicit interaction score based on relationships of the virtual object with other virtual objects, wherein the interaction score is a function of user previous interactions with the virtual object and the implicit interaction score.
6 . The computer program product of claim 5 , wherein the implicit interaction score is calculated from a graph of nodes representing virtual objects and edges between the nodes that captures a relationship of the virtual objects at the nodes on the edges.
7 . The computer program product of claim 1 , wherein the operations further comprise:
processing personal information on the user to compare to metadata on a database of virtual objects to determine virtual objects relevant to the user, wherein the reward scores are calculated for the virtual objects relevant to the user, and wherein the virtual objects assigned to the positions comprise the virtual objects relevant to the user.
8 . The computer program product of claim 1 , wherein the operations further comprise:
receiving, from the mixed reality display, information on a user interaction with a virtual object rendered in the field of view including an identifier of the virtual object, a distance of the user from the virtual object in the field of view, and a change in viewing angle for the user to view and interact with the virtual object; and
storing the information on the user interaction with the virtual object to later use to determine the easiness score indicating an easiness to view the virtual object and frequency of interactions with the virtual object.
9 . A system for determining virtual objects to render in a mixed reality display, comprising:
a processor; and
a computer readable storage medium having computer readable program code embodied therein that when executed by the processor performs operations, the operations comprising:
providing viewability scores for positions in a field of view of a user in which virtual objects are rendered, wherein a viewability score for a position is based on a distance from the user and change in viewing angle to view the position;
calculating reward scores for virtual objects, wherein a reward score for a virtual object is calculated based on an easiness score indicating an easiness to view the virtual object in previous interactions with the virtual object and frequency of interactions with the virtual object;
inputting the reward scores and the viewability scores for the positions into a virtual object placement program to output an assignment of the virtual objects to the positions; and
transmitting the assignment of the virtual objects to the positions to the mixed reality display to cause the mixed reality display to render the virtual objects at the positions in the field of view of the user indicated in the assignment.
10 . The system of claim 9 , wherein the virtual object placement program implements machine learning, wherein the operations further comprise:
receiving feedback indicating interactions with the virtual objects rendered at the positions according to the assignment, wherein the virtual object placement program outputted the assignment with a confidence level indicating a probability that the assignment is optimal;
determining whether the interactions satisfy a positive threshold indicating a high level of interactions with the virtual objects in the positions indicated in the assignment or a low threshold indicating a low level of interactions with the virtual objects in the positions indicated in the assignment;
training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with an increased confidence level comprising an increase to the confidence level in response to determining that the interactions satisfy the positive threshold; and
training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with a decreased confidence level comprising a decrease to the confidence level in response to determining that the interactions do not satisfy the positive threshold.
11 . The system of claim 9 , wherein calculating a reward score for a virtual object further comprises:
calculating an interaction score based on user previous interactions with the virtual object, wherein the reward score is a function of the easiness score and the interaction score.
12 . The system of claim 11 , wherein calculating the interaction score for the virtual object further comprises calculating an implicit interaction score based on relationships of the virtual object with other virtual objects, wherein the interaction score is a function of user previous interactions with the virtual object and the implicit interaction score.
13 . The system of claim 12 , wherein the implicit interaction score is calculated from a graph of nodes representing virtual objects and edges between the nodes that captures a relationship of the virtual objects at the nodes on the edges.
14 . The system of claim 11 , wherein the operations further comprise:
processing personal information on the user to compare to metadata on a database of virtual objects to determine virtual objects relevant to the user, wherein the reward scores are calculated for the virtual objects relevant to the user, and wherein the virtual objects assigned to the positions comprise the virtual objects relevant to the user.
15 . The system of claim 11 , wherein the operations further comprise:
receiving, from the mixed reality display, information on a user interaction with a virtual object rendered in the field of view including an identifier of the virtual object, a distance of the user from the virtual object in the field of view, and a change in viewing angle for the user to view and interact with the virtual object; and
storing the information on the user interaction with the virtual object to later use to determine the easiness score indicating an easiness to view the virtual object and frequency of interactions with the virtual object.
16 . A method for determining virtual objects to render in a mixed reality display, comprising:
providing viewability scores for positions in a field of view of a user in which virtual objects are rendered, wherein a viewability score for a position is based on a distance from the user and change in viewing angle to view the position;
calculating reward scores for virtual objects, wherein a reward score for a virtual object is calculated based on an easiness score indicating an easiness to view the virtual object in previous interactions with the virtual object and frequency of interactions with the virtual object;
inputting the reward scores and the viewability scores for the positions into a virtual object placement program to output an assignment of the virtual objects to the positions; and
transmitting the assignment of the virtual objects to the positions to the mixed reality display to cause the mixed reality display to render the virtual objects at the positions in the field of view of the user indicated in the assignment.
17 . The method of claim 16 , wherein the virtual object placement program implements machine learning, further comprising:
receiving feedback indicating interactions with the virtual objects rendered at the positions according to the assignment, wherein the virtual object placement program outputted the assignment with a confidence level indicating a probability that the assignment is optimal;
determining whether the interactions satisfy a positive threshold indicating a high level of interactions with the virtual objects in the positions indicated in the assignment or a low threshold indicating a low level of interactions with the virtual objects in the positions indicated in the assignment;
training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with an increased confidence level comprising an increase to the confidence level in response to determining that the interactions satisfy the positive threshold; and
training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with a decreased confidence level comprising a decrease to the confidence level in response to determining that the interactions do not satisfy the positive threshold.
18 . The method of claim 16 , wherein calculating a reward score for a virtual object further comprises:
calculating an interaction score based on user previous interactions with the virtual object, wherein the reward score is a function of the easiness score and the interaction score.
19 . The method of claim 18 , wherein calculating the interaction score for the virtual object further comprises calculating an implicit interaction score based on relationships of the virtual object with other virtual objects, wherein the interaction score is a function of user previous interactions with the virtual object and the implicit interaction score.
20 . The method of claim 16 , further comprising:
receiving, from the mixed reality display, information on a user interaction with a virtual object rendered in the field of view including an identifier of the virtual object, a distance of the user from the virtual object in the field of view, and a change in viewing angle for the user to view and interact with the virtual object; and
storing the information on the user interaction with the virtual object to later use to determine the easiness score indicating an easiness to view the virtual object and frequency of interactions with the virtual object.