Relighting fragments for insertion into content
A fragment is relit for insertion into a target scene of an image by obtaining a fragment model for the fragment. A set of detail maps for the fragment model are generated, each of which encodes fine-scale shading effects from the surface detail of the fragment. A target scene model is obtained for the target scene, and the fragment model is inserted into the target scene model. The target scene model with inserted fragment model is rendered, and a composited target scene is generated. A modified target scene is generated by combining the composited target scene and the set of detail maps. Weights assigned to the different detail maps can be changed by the user, allowing the modified target scene to be readily altered without re-rendering the target scene model with the inserted fragment model.
1. A method comprising:
obtaining a fragment model for a fragment;
obtaining a target scene model for a target scene;
generating, based on the fragment model, a set of multiple detail maps for the fragment;
inserting the fragment model into the target scene model;
rendering the target scene model with the inserted fragment model to generate a rendered scene;
generating a composited target scene; and
generating a modified target scene by combining the composited target scene and the set of multiple detail maps.
2. A method as recited in claim 1 , the obtaining the target scene model comprising generating the target scene model from the target scene.
3. A method as recited in claim 1 , the obtaining the fragment model comprising generating the fragment model.
4. A method as recited in claim 1 , the fragment model including a shape model of the fragment and an albedo map for the fragment, and the generating the set of multiple detail maps comprising generating the set of detail maps based on the shape model and the albedo map.
5. A method as recited in claim 1 , the fragment model including a shape model of the fragment, an albedo map for the fragment, and an alpha map for the fragment.
6. A method as recited in claim 1 , the set of detail maps including two or more of: a uniform illumination map, a parametric illumination map, and an inferred illumination map.
7. A method as recited in claim 1 , the generating the composited target scene comprising generating the composited target scene by combining the rendered scene, the target scene, an object mask, and a rendered target scene without inserted fragments.
8. A method as recited in claim 7 , wherein the composited target scene is a composited target scene C, the rendered scene is a rendered scene R, the target scene is a target scene I, the object mask is an object mask M, the rendered target scene without inserted fragments is a rendered target scene E, and the combining comprises:
C=M⊙R +(1− M )⊙( I+R−E ).
9. A method as recited in claim 7 , the generating the modified target scene comprising:
receiving user inputs to change weights applied to the set of multiple detail maps; and
combining the composited target scene and each weighted detail map of the set of multiple detail maps.
10. A method as recited in claim 9 , wherein a first weight coefficient comprises w u , a first weight gamma value comprises γu, a first detail map of the set of multiple detail maps comprises D u , a second weight coefficient comprises w p , a second weight gamma value comprises γp, a second detail map of the set of multiple detail maps comprises D p , a third weight coefficient comprises w i , a third weight gamma value comprises γi, a third detail map of the set of multiple detail maps comprises D i , the modified target scene comprises C final , and the generating the modified target scene comprises:
C final =C+w u D u γu +w p D p γp +w i D i γi .
11. A method as recited in claim 9 , the combining the composited target scene comprising combining the composited target scene without re-rendering the target scene model with the inserted fragment model.
12. One or more computer readable storage media having stored thereon multiple instructions that, when executed by one or more processors of one or more devices, cause the one or more processors to perform acts comprising:
obtaining a fragment model for a fragment;
obtaining a target scene model for a target scene;
generating, based on the fragment model, a set of multiple detail maps for the fragment;
inserting the fragment model into the target scene model;
rendering the target scene model with the inserted fragment model to generate a rendered scene;
generating a composited target scene; and
generating a modified target scene by combining the composited target scene and the set of multiple detail maps.
13. One or more computer readable storage media as recited in claim 12 , the obtaining the target scene model comprising generating the target scene model from the target scene.
14. One or more computer readable storage media as recited in claim 12 , the obtaining the fragment model comprising generating the fragment model.
15. One or more computer readable storage media as recited in claim 12 , the fragment model including a shape model of the fragment and an albedo map for the fragment, and the generating the set of multiple detail maps comprising generating the set of detail maps based on the shape model and the albedo map.
16. One or more computer readable storage media as recited in claim 12 , the fragment model including a shape model of the fragment, an albedo map for the fragment, and an alpha map for the fragment.
17. One or more computer readable storage media as recited in claim 12 , the set of detail maps including two or more of: a uniform illumination map, a parametric illumination map, and an inferred illumination map.
18. One or more computer readable storage media as recited in claim 12 , the generating the composited target scene comprising generating the composited target scene by combining the rendered scene, the target scene, an object mask, and a rendered target scene without inserted fragments.
19. One or more computer readable storage media as recited in claim 18 , wherein the composited target scene is a composited target scene C, the rendered scene is a rendered scene R, the target scene is a target scene I, the object mask is an object mask M, the rendered target scene without inserted fragments is a rendered target scene E, and the combining comprises:
C=M⊙R +(1− M )⊙( I+R−E ).
20. One or more computer readable storage media as recited in claim 18 , the generating the modified target scene comprising:
receiving user inputs to change weights applied to the set of multiple detail maps; and
combining the composited target scene and each weighted detail map of the set of multiple detail maps.
21. One or more computer readable storage media as recited in claim 20 , wherein a first weight coefficient comprises w u , a first weight gamma value comprises γu, a first detail map of the set of multiple detail maps comprises D u , a second weight coefficient comprises w p , a second weight gamma value comprises γp, a second detail map of the set of multiple detail maps comprises D p , a third weight coefficient comprises w i , a third weight gamma value comprises γi, a third detail map of the set of multiple detail maps comprises D i , the modified target scene comprises C final , and the generating the modified target scene comprises:
C final =C+w u D u γu +w p D p γp +w i D i γi .
22. One or more computer readable storage media as recited in claim 20 , the combining the composited target scene comprising combining the composited target scene without re-rendering the target scene model with the inserted fragment model.
23. A system implemented at least in part in hardware, the system comprising:
a modeling module configured to:
obtain a fragment model for a fragment;
obtain a target scene model for a target scene;
generate, based on the fragment model, a set of multiple detail maps for the fragment; and
a rendering and compositing module configured to:
insert the fragment model into the target scene model;
render the target scene model with the inserted fragment model to generate a rendered scene;
generate a composited target scene;
generate a modified target scene by combining the composited target scene and the set of multiple detail maps.