In Shader Graph, "Vertex Displacement", which changes the position of vertices, is a very powerful method that allows you to easily implement dynamic transformations such as flags waving in the wind, waves on the water surface, raised terrain, character morphing, etc. However, if only the vertex coordinates (Position) are transformed and output at the vertex shader stage, serious rendering problems will occur, such as ``Although the shape of the mesh has changed, the way light is reflected and shadows remain flat before transformation.'' In this article, we will thoroughly explain the internal mechanism that causes this lighting flattening bug, and three solutions with practical code to correctly recalculate and reconstruct normals and tangents according to the deformed shape.

1. Root cause: Why does the light not follow when I move the vertex?

Lighting in 3D rendering (such as PBR) is calculated based on the Normal Vector, which indicates which direction the mesh surface is facing, and the Tangent Vector, which indicates the tangent direction.

1. Light reflection angles and normal map sampling all depend on these vectors.

However, in Unity's render pipeline and Shader Graph specifications, even if you input new coordinates into the "Position" port of the Vertex stage and deform the mesh, the normals and tangents stored atvertices are not automatically recalculated.

As a result, the following inconsistencies occur:

  1. Shape deformation: The vertex coordinates are changed and the mesh becomes wavy and uneven.
  2. Lighting calculation: The normal vector ((0, 1, 0), etc.) before the transformation is straight and flat and is sent as is to the pixel shader and used for lighting calculations.
  3. Drawing result: Although the visible three-dimensional effect of the mesh is distorted, only the gradation of shadows is the same as that of a flat board, resulting in an extremely unnatural "paper-pasted" look.

This problem also affects the generation of shadows (Shadow Caster), resulting in a large discrepancy between the outline of the shadow that falls from the edge of the deformed object and the shadow attached to the body (Self-Shadowing). To solve this bug, you need to rotate and transform the normal and tangent in the same way according to the displacement of the vertex (height change and tilt).

2. Solution A: Mathematical recalculation using Analytical Derivatives

If the vertex displacement is defined by a clear mathematical formula (e.g. sine wave or cosine wave), by mathematically differentiating (partial differentiation) the deformed normal and tangent lines can be determined in a pinpoint and ultra-lightweight manner.

For example, assume that the displacement of height $y$ due to a sine wave traveling in the X-axis direction is expressed by the following formula. 

y = sin(x * frequency + time) * amplitude;

In this case, the slope in the X direction (tangent vector) is a cosine wave obtained by differentiating the above equation with $x$. 

dy/dx = frequency * amplitude * cos(x * frequency + time);

From this slope vector, new tangents and normals can be constructed as follows. 

// Define Tangent from the tilt vector in the X direction
float3 tangent = normalize(float3(1.0, dy/dx, 0.0));
// Define Binormal in the Z direction (direction without deformation)
float3 binormal = float3(0.0, 0.0, 1.0);
// Calculate Normal from the cross product
float3 normal = normalize(cross(binormal, tangent));

Since this method does not involve texture sampling or loop processing, it has the great advantage of being extremely low-load and works perfectly in mobile and VR environments.

3. Solution B: Generic reconstruction using finite difference approximation

3D noise (Simplex Noise, etc.) or complex textures, and it is difficult to manually calculate the mathematical differential coefficient, , which samples two virtual adjacent points after deformation and calculates the normal from the difference, is extremely effective.

In this method, in addition to the current vertex position $P$, we assume virtual points $P_{tangent}$ and $P_{bitangent}$ that are separated by a minimal distance $\epsilon$ in the tangent and bitangent directions, and calculate their respective positions after displacement.

Basic principle of normal calculation by difference method:
1. $P' = P + \\text{Displace}(P)$ (Current vertex after deformation)
2. $P_{T}' = (P + \\epsilon \\cdot T) + \\text{Displace}(P + \\epsilon \\cdot T)$ (After deformation of a point away in the Tangent direction)
3. $P_{B}' = (P + \\epsilon \\cdot B) + \\text{Displace}(P + \\epsilon \\cdot B)$ (after deformation of points away in Bitangent direction)
4. $T_{\\text{new}} = P_{T}' - P'$ (new tangent vector)
5. $B_{\\text{new}} = P_{B}' - P'$ (New bitangent vector)
6. $N_{\\text{new}} = \\text{normalize}(\\text{cross}(T_{\\text{new}}, B_{\\text{new}}))$ (New normal vector)

The following is the Shader Graph Custom This is an example HLSL code for vertex displacement and normal/tangent reconstruction using difference approximation that can be incorporated into the Function node. 

// A function that defines vertex deformation (here, a 3D sine wave is used as an example)
float3 CalculateDisplacement(float3 pos, float frequency, float amplitude, float time)
{
 float wave = sin(pos.x * frequency + time) * cos(pos.z * frequency + time);
 return float3(0, wave * amplitude, 0);
}

void ReconstructNormalAndTangent_float(
 float3 objectPosition,
 float3 objectNormal,
 float3 objectTangent,
 float frequency,
 float amplitude,
 float time,
 float epsilon, // Very small distance for difference calculation (0.01 etc. recommended)
 out float3 displacedPosition,
 out float3 reconstructedNormal,
 out float3 reconstructedTangent)
{
 // Calculate Bitangent
 float3 bitangent = cross(objectNormal, objectTangent) * objectTangent.w;

 // 1. Define three points before transformation
 float3 p = objectPosition;
 float3 pT = p + objectTangent * epsilon;
 float3 pB = p + bitangent * epsilon;

 // 2. Apply displacement to each and calculate the transformed coordinates
 float3 displacedP = p + CalculateDisplacement(p, frequency, amplitude, time);
 float3 displacedPT = pT + CalculateDisplacement(pT, frequency, amplitude, time);
 float3 displacedPB = pB + CalculateDisplacement(pB, frequency, amplitude, time);

 // 3. Create new tangent and bitangent from the difference in coordinates after transformation
 float3 newTangent = displacedPT - displacedP;
 float3 newBitangent = displacedPB - displacedP;

 // 4. Calculate new normal vector by cross product
 reconstructedNormal = normalize(cross(newTangent, newBitangent));
 reconstructedTangent = normalize(newTangent);
 displacedPosition = displacedP;
}

4. Steps for connecting nodes on Shader Graph

Connect the created HLSL custom function or calculation node correctly on Shader Graph as shown below.

  1. Custom Function node placement: Create a Custom Function node that defines the `ReconstructNormalAndTangent_float` function above.
  2. Connecting object data: Connect the `Position (Object Space)`, `Normal (Object Space)`, and `Tangent (Object Space)` nodes of the Vertex stage to the corresponding input ports of the Custom Function.
  3. Adjusting parameters: Connect a constant of about `0.01` to the `epsilon` input port, or a minute value depending on the mesh resolution (polygon density).
  4. Output to master stack:
    • Connect the `displacedPosition` output of the Custom Function to the Vertex Position port of the master stack.
    • Connect the `reconstructedNormal` output to the Vertex Normal (or Fragment Normal) port of the master stack.
    • Connect the `reconstructedTangent` output to the master stack of Vertex Tangent ports.

*Note: Normals and tangents must be recalculated basically in the Vertex stage. If calculated in the Fragment stage, there will be a conflict with the interpolation process inside the polygon, causing the highlights to become jagged.