Material System Overview

Material System Upgrade Guide

Cocos Creator has supported the material system since v2.x. In v3.0, the design and built-in Shader API of the material system continues to be improved. When upgrading from v2.x to v3.0 and later versions, some of the content may still need to be adjusted manually by the developer, please refer to the upgrade guide below:

Material System Class Diagram

The material system plays an essential role in any game engine infrastructure, it controls the way everything is drawn on screen and much more.

The general structure of the system is as follows:

Assets

EffectAsset

EffectAsset is a shading procedure description file, written by both engine and game developers. It contains the mathematical calculations and algorithms for calculating the color of each pixel rendered.
When the builtin effects are not the best fit for your need, writing your own effect can give you all the capabilities to customize the rendering process.

Detailed syntax instructions can be found in the Effect Syntax documentation.

Here is the flow that the engine reads EffectAsset resource: when the editor imports EffectAsset, the engine will do a pre-processing on your written content, then replace GL string as constant in the pipeline, extract shader information, convert shader version and so on.

Using builtin-unlit.effect as an example, the structure of the compiled output EffectAsset is roughly as follows:

  1. {
  2. "name": "builtin-unlit",
  3. "techniques": [{
  4. "name": "opaque",
  5. "passes": [{
  6. "program": "builtin-unlit|unlit-vs:vert|unlit-fs:frag",
  7. "properties": {
  8. "mainTexture": {
  9. "value": "grey",
  10. "type": 28
  11. },
  12. "tilingOffset": {
  13. "value": [1, 1, 0, 0],
  14. "type": 16
  15. },
  16. "mainColor": {
  17. "value": [1, 1, 1, 1],
  18. "editor": { "type": "color" },
  19. "type": 16
  20. },
  21. "colorScale": {
  22. "value": [1, 1, 1],
  23. "type": 15,
  24. "handleInfo": ["colorScaleAndCutoff", 0, 15]
  25. },
  26. "alphaThreshold": {
  27. "value": [0.5],
  28. "editor": { "parent": "USE_ALPHA_TEST" },
  29. "type": 13,
  30. "handleInfo": ["colorScaleAndCutoff", 3, 13]
  31. },
  32. "color": {
  33. "editor": { "visible": false },
  34. "type": 16, "handleInfo": ["mainColor", 0, 16]
  35. },
  36. "colorScaleAndCutoff": {
  37. "type": 16,
  38. "editor": { "visible": false, "deprecated": true },
  39. "value": [1, 1, 1, 0.5]
  40. }
  41. },
  42. "migrations": {
  43. "properties": {
  44. "mainColor": { "formerlySerializedAs": "color" }
  45. }
  46. }
  47. }]
  48. }],
  49. "shaders": [{
  50. "name": "builtin-unlit|unlit-vs:vert|unlit-fs:frag",
  51. "hash": 2093221684,
  52. "glsl4": {
  53. "vert": "// glsl 460 vert source, omitted here for brevity",
  54. "frag": "// glsl 460 frag source, omitted here for brevity",
  55. },
  56. "glsl3": {
  57. "vert": "// glsl 300 es vert source, omitted here for brevity",
  58. "frag": "// glsl 300 es frag source, omitted here for brevity",
  59. },
  60. "glsl1": {
  61. "vert": "// glsl 100 vert source, omitted here for brevity",
  62. "frag": "// glsl 100 frag source, omitted here for brevity",
  63. },
  64. "attributes": [
  65. { "tags": ["USE_BATCHING"], "name": "a_dyn_batch_id", "type": 13, "count": 1, "defines": ["USE_BATCHING"], "location": 1 },
  66. { "name": "a_position", "type": 15, "count": 1, "defines": [], "location": 0 },
  67. { "name": "a_weights", "type": 16, "count": 1, "defines": ["USE_SKINNING"], "location": 2 },
  68. { "name": "a_joints", "type": 16, "count": 1, "defines": ["USE_SKINNING"], "location": 3 },
  69. { "tags": ["USE_VERTEX_COLOR"], "name": "a_color", "type": 16, "count": 1, "defines": ["USE_VERTEX_COLOR"], "location": 4 },
  70. { "tags": ["USE_TEXTURE"], "name": "a_texCoord", "type": 14, "count": 1, "defines": ["USE_TEXTURE"], "location": 5 }
  71. ],
  72. "varyings": [
  73. { "name": "v_color", "type": 16, "count": 1, "defines": ["USE_VERTEX_COLOR"], "location": 0 },
  74. { "name": "v_uv", "type": 14, "count": 1, "defines": ["USE_TEXTURE"], "location": 1 }
  75. ],
  76. "builtins": {
  77. "globals": {
  78. "blocks": [
  79. { "name": "CCGlobal", "defines": [] }
  80. ],
  81. "samplers": []
  82. },
  83. "locals": {
  84. "blocks": [
  85. { "name": "CCLocalBatched", "defines": ["USE_BATCHING"] },
  86. { "name": "CCLocal", "defines": [] },
  87. { "name": "CCSkinningTexture", "defines": ["USE_SKINNING", "ANIMATION_BAKED"] },
  88. { "name": "CCSkinningAnimation", "defines": ["USE_SKINNING", "ANIMATION_BAKED"] },
  89. { "name": "CCSkinningFlexible", "defines": ["USE_SKINNING"] }
  90. ],
  91. "samplers": [
  92. { "name": "cc_jointsTexture", "defines": ["USE_SKINNING", "ANIMATION_BAKED"] }
  93. ]
  94. }
  95. },
  96. "defines": [
  97. { "name": "USE_BATCHING", "type": "boolean", "defines": [] },
  98. { "name": "USE_SKINNING", "type": "boolean", "defines": [] },
  99. { "name": "ANIMATION_BAKED", "type": "boolean", "defines": ["USE_SKINNING"] },
  100. { "name": "CC_SUPPORT_FLOAT_TEXTURE", "type": "boolean", "defines": ["USE_SKINNING", "ANIMATION_BAKED"] },
  101. { "name": "USE_VERTEX_COLOR", "type": "boolean", "defines": [] },
  102. { "name": "USE_TEXTURE", "type": "boolean", "defines": [] },
  103. { "name": "FLIP_UV", "type": "boolean", "defines": ["USE_TEXTURE"] },
  104. { "name": "CC_USE_HDR", "type": "boolean", "defines": [] },
  105. { "name": "USE_ALPHA_TEST", "type": "boolean", "defines": [] },
  106. { "name": "ALPHA_TEST_CHANNEL", "type": "string", "defines": ["USE_ALPHA_TEST"], "options": ["a", "r", "g", "b"] }
  107. ],
  108. "blocks": [
  109. {
  110. "name": "TexCoords",
  111. "defines": ["USE_TEXTURE"],
  112. "binding": 0,
  113. "members": [
  114. { "name": "tilingOffset", "type": 16, "count": 1 }
  115. ]
  116. },
  117. {
  118. "name": "Constant",
  119. "defines": [],
  120. "binding": 1,
  121. "members": [
  122. { "name": "mainColor", "type": 16, "count": 1 },
  123. { "name": "colorScaleAndCutoff", "type": 16, "count": 1 }
  124. ]
  125. }
  126. ],
  127. "samplers": [
  128. { "name": "mainTexture", "type": 28, "count": 1, "defines": ["USE_TEXTURE"], "binding": 30 }
  129. ]
  130. }
  131. ]
  132. }

There is a lot to unpack here, but for the most part the details won’t be of any concern to game developers, and the key insight you need to remember is:

  • All the necessary info for runtime shading procedure setup on any target platform (and even editor support) is here in advance to guarantee portability and performance.
  • Redundant info will be trimmed at build-time to ensure minimum space consumption.

Material

Material defines how a surface should be rendered, by including references to textures it uses, tiling information, color tints and more.

The available options for a Material depend on which EffectAsset it is using. Essential parameters for setting up a Material object are:

  • effectAsset or effectName: Effect reference, specifying which EffectAsset will be used (must specify).
  • technique: Inside the EffectAsset, specifying which technique will be used, default to 0.
  • defines: The list of macros, specify what value the shader macros have (for shader variants), default all to 0 (disabled), or in-shader specified default value.
  • states: If any, specifying which pipeline state to override (default to nothing, keep everything the same as how they are specified in effect)
  1. const mat = new Material();
  2. mat.initialize({
  3. effectName: 'pipeline/skybox',
  4. defines: { USE_RGBE_CUBEMAP: true }
  5. });

With this information, the Material can be properly initialized, indicated by the generation of an array of Pass objects for rendering, which can be used for rendering specific models.

Knowing which EffectAsset is currently using, we can specify all the shader properties:

  1. mat.setProperty('cubeMap', someCubeMap);
  2. console.log(mat.getProperty('cubeMap') === someCubeMap); // true

These properties are assigned inside the material, which is just an asset by itself, and hasn’t connected to any model.

To apply the material on a specific model, it needs to be attached to a RenderableComponent. Any component that accepts a material parameter (MeshRenderer, SkinnedMeshRenderer, etc.) is inherited from it.

  1. const comp = someNode.getComponent(MeshRenderer);
  2. comp.material = mat;
  3. comp.setMaterial(mat, 0); // same as last line

According to the number of sub-models, RenderableComponent may reference multiple Material:

  1. comp.setMaterial(someOtherMaterial, 1); // assign to second sub-model

The same Material can be attached to multiple RenderableComponent too:

  1. const comp2 = someNode2.getComponent(MeshRenderer);
  2. comp2.material = mat; // the same material above

When one of the material-sharing models needs to customize some property, you need to get a copied instance of the material asset, aka. MaterialInstance, from the RenderableComponent, by calling:

  1. const mat2 = comp2.material; // copy constructor, now 'mat2' is an 'MaterialInstance', and every change made to `mat2` only affect the 'comp2' instance

The biggest difference between Material asset and MaterialInstance is: MaterialInstance is definitively attached to one RenderableComponent at the beginning of its life cycle, while Material has no such limit.

For MaterialInstances it is possible to modify shader macros or pipeline states:

  1. mat2.recompileShaders({ USE_EMISSIVE: true });
  2. mat2.overridePipelineStates({ rasterizerState: { cullMode: GFXCullMode.NONE } });

Updating shader properties every frame is a common practice, under situations like this, where performance matters, use lower level APIs:

  1. // Save these when starting
  2. const pass = mat2.passes[0];
  3. const hColor = pass.getHandle('albedo');
  4. const color = new Color('#dadada');
  5. // inside update function
  6. color.a = Math.sin(director.getTotalFrames() * 0.01) * 127 + 127;
  7. pass.setUniform(hColor, color);

And for any other changes (different effect, technique, etc.) you have to create a new material from scratch and assign it to the target RenderableComponent.

Built-in materials

Although the material system itself doesn’t make any assumptions on the content, there are some built-in effects written on top of the system, provided for common usage: unlit, physically-based (standard), skybox, particle, sprite, etc.

For a quick reference, here is how each shading term in builtin-standard will be assembled from input data:

Standard

Here are the complete list of properties and macros for it:

PropertyInfo
tilingOffsettiling and offset of the model UV, xy channel for tiling, zw channel for offset
albedo/mainColoralbedo color, the main base color of the model
albedoMap/mainTexturealbedo texture, if present, will be multiplied by the albedo property
albedoScalealbedo scaling factor
weighting the whole albedo factor before the final output
alphaThresholdtest threshold for discarding pixels, any pixel with target channel value lower than this threshold will be discarded
normalMapnormal map texture, enhancing surface details
normalStrenthstrenth of the normal map, the bigger the bumpier
pbrMap
R (AO)
G (Roughness)
B (Metallic)
PBR parameter all-in-one texture: occlusion, roughness and metallic
sample result will be multiplied by the matching constants
metallicRoughnessMap
G (Roughness)
B (Metallic)
metallic and roughness texture
sample result will be multiplied by the matching constants
occlusionMapindependent occlusion texture
sample result will be multiplied by the matching constants
occlusionocclusion constant
roughnessroughness constant
metallicmetallic constant
emissiveemissive color
emissiveMapemissive color texture, if present, will be multiplied by the emissive property,
so remember to set emissive property more close to white
(default black) for this to take effect
emissiveScaleemissive scaling factor
weighting the whole emissive factor before the final output

Accordingly, these are the available macros:

MacroInfo
USE_BATCHINGWhether to enable dynamic VB-merging-style batching
USE_INSTANCINGWhether to enable dynamic instancing
HAS_SECOND_UVWhether there is a second set of UV
ALBEDO_UVSpecifies the uv set to use when sampling albedo texture, default to the first set
EMISSIVE_UVSpecifies the uv set to use when sampling emissive texture, default to the first set
ALPHA_TEST_CHANNELSpecifies the source channel for alpha test, default to A channel
USE_VERTEX_COLORIf enabled, vertex color will be multiplied to albedo factor
USE_ALPHA_TESTWhether to enable alpha test
USE_ALBEDO_MAPWhether to enable albedo texture
USE_NORMAL_MAPWhether to enable normal map
USE_PBR_MAPWhether to enable PBR parameter 3-in-1 texture
As per the glTF spec, the RGB channels must correspond to occlusion, roughness and metallicity respectively
USE_METALLIC_ROUGHNESS_MAPWhether to enable metallic-roughness texture
As per the glTF spec, the GB channels must correspond to roughness and metallicity respectively
USE_OCCLUSION_MAPWhether to enable occlusion texture
Only the red channel will be used, as per glTF spec
USE_EMISSIVE_MAPWhether to enable emissive texture