Rapid generation of 3D heads with natural language
Two dimensional images are converted to a 3D neural radiance field (NeRF), which is modified based on text input to resemble the type of character demanded by the text. An open-source “CLIP” model scores how well an image matches a line of text to produce a final 3D NeRF, which may be converted to a polygonal mesh and imported into a computer simulation such as a computer game.
1. A device comprising:
at least one computer storage that is not a transitory signal and that comprises instructions executable by at least one processor system to:
generate a base three dimensional (3D) neural radiance field (NeRF) from plural images;
use text input to a Contrastive Language-Image Pre-training (CLIP) model to generate a modified NeRF from the base 3D NeRF; and
convert the modified NeRF to a polygonal mesh representing a virtual human head for presentation of the virtual human head in at least one computer simulation; wherein the instructions are executable to:
use a machine learning (ML) model on the base 3D NeRF to minimize a loss indication in matching the text;
train the ML model on a chain of causality from initial image parameters that control vertices of an object to pixels of the object rendered onscreen.
2. The device of claim 1 , wherein the CLIP model rates an image match to the text.
3. The device of claim 2 , wherein the CLIP model is trained on image-text pairs using cosine similarity to score a goodness of match.
4. The device of claim 1 , wherein the ML model comprises at least one fully connected (non-convolutional) deep network.
5. The device of claim 1 , wherein input to the ML model comprises values representing three spatial dimensions and two viewing dimensions.
6. The device of claim 5 , wherein output of the ML model comprises volume density and view-dependent emitted radiance.
7. The device of claim 1 , wherein the instructions are executable to:
generate the text from a starting phrase using learned ensuing phrases.
8. The device of claim 1 , comprising the at least one processor.