IP Library Granted Patent US 9,998,705
Granted Patent B2
US 9,998,705 · App. 15/231,626 · Granted Jun 12, 2018

Hybrid visual communication

Inventor: Schuyler Cullen (San Jose, CA)
Assignee: Samsung Electronics Co., Ltd.
H04N7/141G06K9/00221G06T7/10G06T7/194G06T7/70H04N7/147G06K2209/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,998,705
App. No.
15/231,626
Granted
Jun 12, 2018
Kind
B2
Abstract

Exemplary embodiments for visual communication between a first device and a second device, comprising: creating a 3D mesh model of a first device user; receiving sensor data from a sensor array during the visual communication session between the first device and the second device, wherein the image data includes motion of the first device user; determining 3D mesh model updates using the image data; transmitting the 3D mesh model updates to the second device for the second device to update display of the 3D mesh model of the first device user, wherein the update is represented as one or more of a blend shape and a relative vertex position change of the 3D mesh model.

Claims (42)

1. A method for visual communication between a first device and a second device, comprising:

creating a first three-dimensional (3D) mesh model of a first device user;

receiving sensor data from one or more sensors on the first device during the visual communication session between the first device and the second device, wherein the sensor data includes image data;

determining first 3D mesh model updates using the image data; and

transmitting the first 3D mesh model updates to the second device for the second device to provide a first updated display of the first 3D mesh model of the first device user on the second device, wherein the first 3D mesh model updates are represented as one or more of a blend shape and a relative vertex position change of the first 3D mesh model;

transmitting a common background image to the second device so that the first 3D mesh model is superimposed over the common background image when displayed on the second device during the visual communication session, wherein the common background image comprises one of a first background image from the image data of the first device user, and a second pre-determined background image; and

wherein responsive to a switch from a 3D visual mode to a 2D video mode on the first device, gradually blending and transmitting a stream of the first 3D mesh model updates with a 2D video stream to the second device until only the 2D video stream is transmitted.

2. The method of claim 1 , further comprising generating an avatar face based on the 3D mesh model.

3. The method of claim 1 , further comprising encrypting the first 3D mesh model updates.

4. The method of claim 1 , further comprising selecting the common background image as one of the first background image and the second pre-determined background image based on user configuration.

5. The method of claim 1 , further comprising:

receiving a second 3D mesh model from the second device; and

superimposing the second 3D mesh model over the common background image when displayed on the first device during the visual communication session.

6. The method of claim 1 , further comprising:

arranging the first 3D mesh model and the second 3D mesh model in a desired arrangement in real-time.

7. The method of claim 6 , further comprising:

superimposing the first 3D mesh model and the second 3D mesh model in the desired arrangement over the common background image in the desired arrangement when displayed on one or more of the first device and the second device during the visual communication session.

8. The method of claim 6 , further comprising:

superimposing the first 3D mesh model and the second 3D mesh model in the desired arrangement over the common background image; and

transmitting the superimposed first 3D mesh model and the second 3D mesh model in the desired arrangement over the common background image to a third device.

9. The method of claim 1 , further comprising providing one or more of compression and encoding of the first 3D mesh model.

10. An executable software product stored on a non-transitory computer-readable storage medium containing program instructions for providing a visual communication session between a first device and a second device, the program instructions to perform operations including:

create a first three-dimensional (3D) mesh model of a first device user;

receive sensor data from one or more sensors on the first device during the visual communication session between the first device and the second device, wherein the sensor data includes image data;

determine first 3D mesh model updates using the image data; and

transmit the first 3D mesh model updates to the second device for the second device to provide a first updated display of the first 3D mesh model of the first device user on the second device, wherein the first 3D mesh model updates are represented as one or more of a blend shape and a relative vertex position change of the first 3D mesh model;

wherein the program instructions further transmit a common background image to the second device so that the first 3D mesh model is superimposed over the common background image when displayed on the second device during the visual communication session, wherein the common background image comprises one of: a first background image from the image data of the first device user, and a second pre-determined background image; and

wherein responsive to a switch from a 3D visual mode to a 2D video mode on the first device, gradually blending and transmitting a stream of the first 3D mesh model updates with a 2D video stream to the second device until only the 2D video stream is transmitted.

11. The executable software product stored on a computer-readable storage medium of claim 10 , wherein the program instructions further generate an avatar face based on the 3D mesh model.

12. The executable software product stored on a computer-readable storage medium of claim 10 , wherein the program instructions further encrypt the first 3D mesh model updates.

13. The executable software product stored on a computer-readable storage medium of claim 10 , wherein the program instructions further select the common background image as one of the first background image and a second pre-determined background image based on user configuration.

14. The executable software product stored on a computer-readable storage medium of claim 10 , wherein the program instructions further:

receive a second 3D mesh model from the second device; and

superimpose the second 3D mesh model over the common background image when displayed on the first device during the visual communication session.

15. The executable software product stored on a computer-readable storage medium of claim 10 , wherein the program instructions further:

arrange the first 3D mesh model and the second 3D mesh model in a desired arrangement in real-time.

16. The executable software product stored on a computer-readable storage medium of claim 15 , wherein the program instructions further:

superimpose the first 3D mesh model and the second 3D mesh model in the desired arrangement over the common background image when displayed on one or more of the first device and the second device during the visual communication session.

17. The executable software product stored on a computer-readable storage medium of claim 15 , wherein the program instructions further:

superimpose the first 3D mesh model and the second 3D mesh model in the desired arrangement over the common background image; and

transmit the superimposed first 3D mesh model and the second 3D mesh model in the desired arrangement over the common background image to a third device.

18. The executable software product stored on a computer-readable storage medium of claim 10 , wherein the program instructions further provide one or more of compression and encoding of the first 3D mesh model.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 8, 2016
From: CULLEN, SCHUYLER
To: SAMSUNG ELECTRONICS CO., LTD.
Reel/Frame 039373/0364 →
Continuity (4)
Continuation In Part 15073632 · Mar 17, 2016
Continuation 14052803 · Oct 14, 2013
Provisional Application 61864446 · Aug 9, 2013
Related Publication 20160353056A1 · Dec 1, 2016