An alternative set of eyes with support for glTF/GLB export

[met4citizen]

In web apps using our TalkingHead module, the camera view is often a close-up of the character's face, so the eyes are a very prominent feature. Unfortunately, Blender has very limited support for shader nodes that can be exported to the glTF/GLB format. Therefore MPFB's procedural eyes, for example, cannot be used. And while MakeSkin/GameEngine eyes can be made compatible with small shader setup changes (removing the current color mix node that is not supported), the result is not as good visually. In addition, the current eye textures are quite large for web app use, typically >600kB each.

To address these issues, I have tried to find a good compromise between high quality, glTF/GLB compatibility, and small file size.

As part of my experimentation, I created a new set of eyes in a few different colors. I used the existing eyeball texture as a starting point, replaced the irises, added normal maps, added a reflective material for the cornea, and designed a simple shader node setup compatible with glTF/GLB export.

In addition, to decrease the exported file size, I used the WebP image format instead of PNG. I used a low compression rate so the quality remains high, but the file size is now only around 100 kB in total for diffuse + normal. The compression could, of course, be done also when actually exporting, or by using a tool like glTF_Transform after the export.

Here are a couple of examples for brown and light blue eyes at the moment. First is the current MakeSkin eyes for comparison, followed by the new glTF/GLB-version.

Here is my simple shader setup for the eyes. The "Math" and "Normal Map" nodes used here are supported by the Blender glTF exporter. The exporter does not support any color mix nodes, so each eye color must have its own separate diffuse file:

In addition to the above, I have a separate transparent material for the cornea.

Now I'm trying to find the best way to share these glTF/GLB-compatible eyes with our TalkingHead developers as easy-to-use assets. I haven't created any eyes, clothes, or materials using MPFB before, so I was wondering whether the two-material shader setup (eyes + cornea) can be created using .mhmat files, or if that would require code changes?

Any ideas or suggestions on the best approach going forward with this?

[joepal1976]

The short answer is that MHMAT does not support multiple materials in one file. It supports a list of textures as well as global object settings for roughness etc.

Without discussing technical solution for defining the material setup: how you do know which verts belong to the cornea and which verts belong to the rest of the eye?

I can see a few ways forward:

• Supporting another eye material model would require a code change, but isn't a huge deal. Especially not if it can be assumed to be useful for a wider set of users. You could, for example, use the alpha channel to decide if the material should be assumed to be cornea.
• Maybe the cornea and the eyeball does not need to be the same object? If it is acceptable to split them, there's no problem in having different mhmat definitions for them. Downside is you would need to load two objects for eyes to be complete.

[met4citizen]

Thank you. I appreciate the input and suggestions.

At the moment, it seems that some of the transparency settings for the cornea that look good in Blender do not render well in Three.js, and vice versa. I need to investigate this further, but it may turn out that I have to implement a specific solution that might not be suitable for a wider set of users.

That said, adding things like corneas and support for multiple materials could still be a useful feature for MPFB, so I will write down some further comments and ideas here.

Maybe the cornea and the eyeball does not need to be the same object?

Yes, this would be one solution, but as you said, it has some downsides. For one thing, runtime performance is presumably better with a single mesh using multiple materials than with separate meshes each using one material. A single mesh means fewer objects, fewer matrix updates, fewer frustum-culling checks, etc. Of course in practice the difference becomes noticeable only when we are talking about hundreds or thousands of objects. Nevertheless, since the cornea should naturally move with the eyeball, I think it makes sense to keep them as the same object.

how you do know which verts belong to the cornea and which verts belong to the rest of the eye?

My current understanding is that this information can be stored in the OBJ file. It just requires the export_material_groups=True option in obj_export. When the OBJ file gets imported using obj_import, mesh faces get mapped to material slots automatically.

I actually tested this. First, I manually exported my new eyes to OBJ with material groups and MTL materials. I then removed the original "material" entry from the mhclo file. When I added the eyes in MPFB, I got the eyeballs and the cornea in two separate material slots.

However, when the mhclo file has a "material" entry specified, the MPFB code (presumably) clears all previously imported MTL materials and material slots and adds a new material based on the mhmat file. In the process, the mapping between mesh faces and material slots is lost.

Instead of resetting the imported OBJ setup, perhaps the code could simply override the settings of the existing material slot and build the shader node setup there. Also, the "material" entry in the mhclo file could potentially accept multiple mhmat file names that would be configured in order, starting from the first slot.

Note that I haven't traced all the steps how this is actually done in the code, so at this point I'm merely speculating.

[joepal1976]

That's actually pretty interesting. Supporting multiple materials for one mesh would be useful.

If the basic support is already there in wavefront, I guess the two major things to investigate would be a) supporting these in markup and import and b) tooling for creating assets.

The second isn't necessarily required to achieve the first. To begin with, manually modified descriptors could be sufficient.

Supporting the correct way of achieving transparency would still be needed obviously.

[met4citizen]

As part of my experimentation, I separated the sclera (if that is the correct term) so that it has its own material. This made it possible to remove the Alpha entirely from the eyeball, thereby simplifying the shader setup enough to allow exporting and importing using only the Wavefront OBJ/MTL format. As a result, there are now three materials, and the reflective properties of the sclera and cornea can be adjusted independently.

To test this, I first manually exported my eyes to OBJ/MTL, selecting UVs, Normals, Material Groups, and PBR Extensions. I then removed the material entry from the MHCLO file and copied all the required files to the user library folder ./eyes/talkinghead. Here are the zipped files, if you would like to take a look:

talkinghead-eyes-brown.zip

When I add these new "talkinghead" eyes to a character in MPFB, the materials and shader setup appear as expected. For best results in Blender, I only needed to set the cornea material Settings | Surface | Render Method to "Blended" (and possibly set the thickness to "Slab"). For best results in Three.js, I only needed to increase the cornea's Alpha value slightly to get better reflections.

This approach is, of course, somewhat of a hack, as it does not use MHMAT materials and lacks some nice MPFB features, such as the ability to use alternative materials.

In general, I would assume that adding support for multiple materials in the code would be possible with a reasonable amount of effort. Unfortunately, my calendar is currently full. However, if you (or anyone else) decide to work on this , I can try to find some time for testing.

[joepal1976]

In general, I would assume that adding support for multiple materials in the code would be possible with a reasonable amount of effort. Unfortunately, my calendar is currently full. However, if you (or anyone else) decide to work on this , I can try to find some time for testing.

Now worries, I'll build it. It will be a bit down the prio list, but I'll get there eventually.