Advanced systems and methods for automatically generating an animatable object from various types of user input

1. A method for creating a customized animatable 3D model from a combination of input devices and sensors, the method comprising:

receiving a first input from a first device and a second input from the first device or from at least one other device, each of the first input and the second input comprising a point cloud of an image associated with a face or a head of a user, the first input being associated with at least one sensor on the first device and the second input being associated with the at least one sensor on the first device, at least one different sensor on the first device, or at least one different sensor on the at least one other device;

preprocessing the first input and the second input, the preprocessing comprising determining orientation of the point clouds for the first input and the second input, the orientation of the point clouds of the image in space being unknown based on the first and the second inputs as the at least one sensor on the first device or on the at least one other device is in an unknown position regarding location with respect to the face the user and the determining the orientation for the first input and the second input using photogrammetry includes quantifying each of the point clouds of the images from the first and the second inputs regarding location with respect to the face the user;

registering by means of rigid or non-rigid registration the point clouds from the first and the second inputs to align the preprocessed point clouds to a common reference; and

after registering:

automatically detecting one or more features of the registered point clouds;

deforming a template geometry based on the one or more detected features to automatically generate a custom geometry;

determining a texture of the received first and second inputs and transferring the texture to the custom geometry;

deforming a template control structure based on the one or more detected features to automatically generate a custom control structure; and

generating an animatable object having the custom geometry, the transferred texture, and the custom control structure.

2. The method of claim 1 , whether texture data is included in the first input and/or the second input, the method further comprising determining the texture comprising using the included texture data for determining depth values for different positions.

3. The method of claim 1 , wherein the orientation of the point clouds in space from the first input and the second input are provided in the first input and the second input.

4. The method of claim 1 , further comprising estimating the orientation in space of the point cloud from the second input.

5. The method of claim 4 , wherein the second input from the at least one other device includes at least one two-dimensional image.

6. The method of claim 1 , wherein the at least one sensor of the first device includes a dot projection sensor and/or a Time of Flight sensor; wherein the at least one sensor of the first device provides low resolution texture data or lacks texture data.

7. The method of claim 1 , wherein the at least one other device provides high quality texture data and includes at least one camera rig, at least one depth sensor, or at least one structured light sensor (SLS), or a combination thereof.

8. The method of claim 1 , wherein the first input and the second input each includes at least one of an image, a video signal, an audio signal, and a 3D scan.

9. The method of claim 1 , wherein the transferring the texture to the custom geometry includes automatically mapping at least one pixel of the texture to a corresponding vertex on the custom geometry.

10. The method of claim 1 , wherein the automatically detecting the one or more features includes determining a set of spatial coordinates, each spatial coordinate associated with one of the one or more features of the received first and second inputs.

11. The method of claim 1 , wherein a set of features of the template geometry corresponds to the one or more detected features, the deforming the template geometry including matching the one or more detected features to the set of features of the template geometry.

12. The method of claim 11 , wherein the deforming the template geometry to automatically generate the custom geometry uses radial basis functions.

13. The method of claim 1 , wherein a set of control elements of the template control structure corresponds to a subset of the one or more detected features including a bone-based rig, a viseme set, or a blend shape, the deforming the template control structure including matching the subset of the one or more detected features to the set of control elements of the template control structure.

14. The method of claim 1 , wherein the method runs on a server coupled via at least one network to the first device and the at least one other device.

15. The method of claim 1 , wherein the at least one other device includes one or more of a camera, smart phone, depth sensor, time of flight sensor, a device that can scan in three dimensions (3D), and structured light sensor, or combinations thereof.

16. A system for automatically generating an animatable object, the system comprising:

a processor; and

a memory for storing executable instructions, the processor executing the instructions to:

receive a first input from a first device and a second input from the first device or from at least one other device, each of the first and the second input comprising a point cloud of an image associated with a face or head of a user, the first input being associated with at least one sensor on the first device and the second input being associated with the at least one sensor on the first device, at least one different sensor on the first device, or at least one different sensor on the at least one other device;

preprocess the first and the second input, comprising:

determining orientation of the point clouds for the first input and the second input, the orientation of the point clouds of the image in space being unknown based on the first and the second inputs as the at least one sensor on the first device or on the at least one other device is in an unknown position regarding location with respect to the face of the user and the determining the orientation for the first input and the second input using photogrammetry includes quantifying each of the point clouds of the images from the first and the second inputs regarding location with respect to the face the user;

register by means of rigid or non-rigid registration the point clouds from the first and the second inputs to align the preprocessed point clouds to a common reference; and

after registering:

automatically detect one or more features of the registered point clouds;

deform a template geometry based on the one or more detected features to automatically generate a custom geometry;

determine a texture of the received first and second inputs and transferring the texture to the custom geometry;

deform a template control structure based on the one or more detected features to automatically generate a custom control structure; and

generate an animatable object having the custom geometry, the transferred texture, and the custom control structure.

17. The system of claim 16 , wherein vertices of the custom geometry are based on spatial coordinates determined from the one or more detected features of the received first and the second input; and wherein control elements of the custom control structure are based on spatial coordinates determined from the one or more detected features of the received first and the second input.

18. The system of claim 16 , wherein the automatically detecting the one or more features, the deforming the template geometry, the transferring the texture, and the deforming the template control structure are being performed in real-time.

19. A system for creating a customized animatable 3D model from a combination of input devices and sensors, the system comprising:

means for receiving a first input from a first device and a second input from the first device or from at least one other device, each of the first input and the second input comprising a point cloud of an image associated with a face or a head of a user, the first input being associated with at least one sensor on the first device and the second input being associated with the at least one sensor on the first device, at least one different sensor on the first device, or at least one different sensor on the at least one other device;

means for preprocessing the first input and second input, the preprocessing comprising:

determining orientation of the point clouds for the first input and the second input, the orientation of the point clouds of the image in space being unknown based on the first and second inputs as the at least one sensor on the first device or on the at least one other device is in an unknown position regarding location with respect to the face of the user and the determining the orientation for the first input and the second input using photogrammetry includes quantifying each of the point clouds of the images from the first and the second inputs regarding location with respect to the face of the user;

means for registering the point clouds from the first and the second inputs to align the preprocessed point clouds to a common reference; and

after registering:

means for automatically detecting one or more features of the registered point clouds;

means for deforming a template geometry based on the one or more detected features to automatically generate a custom geometry;

means for determining a texture of the received first and second inputs and transferring the texture to the custom geometry;

means for deforming a template control structure based on the one or more detected features to automatically generate a custom control structure; and

means for generating an animatable object having the custom geometry, the transferred texture, and the custom control structure.