Method and system of determing user engagement and sentiment with learned models and user-facing camera images
In one exemplary embodiment, a method includes the step of obtaining a digital image of a user with a user-facing camera of a computing device. It is determined that the digital image includes a frontal image of the user. A user-sentiment score is calculated based on at least one attribute of the frontal image. A user engagement value is determined with respect to a portion of a display of the computing device. At least one of the frontal image of the user, the user-sentiment score or the gaze position of the user is communicated to an external server process or an application operating in the computing device.
1. A method comprising:
obtaining a digital image of a user with a user-facing camera of a computing device;
determining that the digital image comprises a frontal image of the user;
calculating a user-sentiment score based on at least one attribute of the frontal image;
determining a user engagement value with respect to a portion of a display of the computing device;
communicating at least one of the frontal image of the user, the user-sentiment score or the gaze position of the user to an external server process or an application operating in the computing device;
wherein the step of determining that the digital image comprises a frontal image of the user further comprises:
receiving a frontal-image learned model; and
comparing the digital image with the frontal-image learned model;
wherein the step of determining that the digital image comprises a frontal image of the user further comprises:
determining a sufficient similarity between the digital image and the frontal-image learned model;
wherein the step of calculating a user-sentiment score based at least one attribute of the frontal image further comprises:
receiving a set of learned-smile models, wherein each learned-smile model is associated with a specified user-sentiment score; and
extracting a user-smile attribute from the frontal-image;
wherein the step of calculating a user-sentiment score based on at least one attribute of the frontal image further comprises:
comparing the user-smile attribute with the set of learned-smile models;
determining a best-fit learned-smile model of the set of learned-smile models for the user-smile attribute; and
assigning the user sentiment score the specified user-sentiment score of the best-fit learned-smile model; and
wherein the step of determining a user engagement value with respect to a portion of a display of the computing device further comprises:
mapping a gaze position of the user with respect to a location of the display of the computing device.
2. The method of claim 1 , wherein the computing device comprises a tablet computer.
3. The method of claim 1 further comprising:
detecting that a subsequent digital image comprises a profile image of the user after a previous digital image comprises the frontal image of the user.
4. The method of claim 3 , wherein the form of human-computer interaction comprises:
pausing a play process of a digital video.
5. A computer system comprising:
a processor configured to execute instructions;
a memory containing instructions when executed on the processor, causes the processor to perform operations that:
obtain a digital image of a user with a user-facing camera of a computing device, and wherein the computing device comprises a tablet computer;
determine that the digital image comprises a frontal image of the user;
calculate a user-sentiment score based on at least one attribute of the frontal image, and wherein the operation to calculate a user-sentiment score based on at least one attribute of the frontal image further comprises;
receive a set of learned-smile models, wherein each learned-smile model is associated with a specified user-sentiment score, and
extract a user-smile attribute from the frontal-image
determine a user engagement value with respect to a portion of a display of the computing device, and wherein the operation to determine that the digital image comprises a frontal image of the user further comprises:
receive a frontal-image learned model;
compare the digital image with the frontal-image learned model; and
determine a sufficient similarity between the digital image and the frontal-image learned model; and
communicate at least one of the frontal image of the user, the user-sentiment score or the gaze position of the user to an external server process or an application operating in the computing device.
6. The computer system of claim 5 , wherein the operation to calculate a user-sentiment score based on at least one attribute of the frontal image further comprises;
compare the user-smile attribute with the set of learned-smile models;
determine a best-fit learned-smile model of the set of learned-smile models for the user-smile attribute; and
assign the user sentiment score the specified user-sentiment score of the best-fit learned-smile model.
7. The computer system of claim 5 , wherein the operation to determine a user engagement value with respect to a portion of a display of the computing device further comprises:
map a gaze position of the user with respect to a location of the display of the computing device.
8. The computer system of claim 5 further comprising:
detect that a subsequent digital image comprises a profile image of the user after a previous digital image comprises the frontal image of the user.
9. The computer system of claim 8 , wherein the form of human-computer interaction comprises:
pausing a play process of a digital video.
10. A method comprising:
obtaining, with at least one processor, a digital image of a user with a user-facing camera of a computing device;
determining that the digital image comprises a frontal image of the user;
calculating a user-sentiment score based on at least one attribute of the frontal image;
determining a user engagement value with respect to a portion of a display of the computing device;
communicating at least one of the frontal image of the user, the user-sentiment score or the gaze position of the user to an external server process or an application operating in the computing device;
wherein the step of determining that the digital image comprises a frontal image of the user further comprises:
receiving a frontal-image learned model; and
comparing the digital image with the frontal-image learned model;
wherein the step of determining that the digital image comprises a frontal image of the user further comprises:
determining a sufficient similarity between the digital image and the frontal-image learned model;
wherein the step of calculating a user-sentiment score based on at least one attribute of the frontal image further comprises:
receiving a set of learned-user-sentiment models, wherein each learned-smile model is associated with a specified user-sentiment score; and
extracting a user-sentiment attribute from the frontal-image;
wherein the step of calculating a user-sentiment score based on at least one attribute of the frontal image further comprises:
comparing the user-sentiment attribute with the set of learned-user-sentiment models;
determining a best-fit learned-user-sentiment model of the set of learned-user-sentiment models for the user-sentiment attribute; and
assigning the user sentiment score the specified user-sentiment score of the best-fit learned-user-sentiment model; and
wherein the step of determining a user engagement value with respect to a portion of a display of the computing device further comprises:
mapping a gaze position of the user with respect to a location of the display of the computing device.