Building a Procedural Koi Pond in WebGL

You may have come across this video at some point in time. When I did, I immediately knew that it was going to be the basis for my next personal website. A lot of the procedural animation techniques are copied taken from that video. I'd suggest giving it a watch! It is very well made, and goes into more depth on how inverse kinematics can be applied to animate legs as well.

Rendering the Fish#

As in the video, the shape of the fish is generated using a sequence of circles on a line. The contour is then generated by taking the outermost points on the perimeter.

Smoothing#

Taking these points by themselves produces a jagged line. We can take improve this by smoothing out the fish body with Catmull-Rom curves. Catmull-Rom curves, like Bezier curves, use points to define the actual shape of the curve. However, Catmull-Rom curves are useful because they ensure that the points used to define the curve end up on the curve itself.

Unfortunately, the video's source code uses Processing. While Processing's HTML canvas-like API has its benefits for general use and fast iteration, it is not the most efficient.¹ To implement our koi pond as efficiently as possible, we need to harness the lower-level capabilities of WebGL.

WebGL, the web version of OpenGL, is a low to mid-level graphics API that is the mainstream method of high performance rendering on the web. Simplified, it takes in arrays of triangles, each containing 3 vertices, and renders them to the screen using vertex shaders (that determine where the vertex goes on the screen) and fragment shaders (what color the pixels between vertices should be). So, we need to break our fish mesh down into triangles: through a process called triangularization.

Triangularization#

For general polygons, one method for triangularization is called ear clipping. Ear clipping repeatedly identifies a triangle whose corner "pokes out" of the polygon, and removes it, labeling it as a triangle. If the polygon has vertices, then we must identify at least triangles, each of which requires a linear ² check to see if it truly "sticks out" of the polygon.

This results in a runtime of , meaning a doubling in the number of vertices corresponds to a quadrupling of the running time. For small this is okay. However, because the fish's body is dynamic, we would be required to run this algorithm every single frame. We can do a lot better than this.

Once again, the optimization lies in pushing past generalization. We can clearly identify a property of our use-case that allows for optimization: the fish body is formed from a spine, with points that extend perpendicular to the spine forming the contour. See if you can identify a potential triangularization before clicking on the figure below. Hint: it doesn't change depending on the shape of the fish.

I wont go too much in depth on how exactly to animate the fish (as well as add eyes, fins, and a mouth) as I think the video linked at the start does it better. But once all that is implemented, you have a fully animated fish!

Procedural Koi Patterns#

Our fish is currently looking a little bare. Lets change that.

I mentioned fragment shaders a while back: they are shaders that determine the color of each pixel our mesh occupies on the screen. Right now, our fragment shader is just returning a solid color. However, we can make the fragment shader do a wide variety of things, including but not limited to:

1. coloring parts of the fish depending on their positions in the world
2. coloring parts of the fish depending on their position in relation to the fish
3. sampling a texture to color parts of the fish

In particular, we are interested in number 2. For example, we would want to make certain parts hear the head of a koi fish red. We would want certain parts near the tail to be dark blue. The color we output depends on where that part of the body is.

Noise#

Fortunately for me (and you), there already exists literature³ on how to generate realistic looking koi patterns with shaders. These methods involve noise functions/maps, which allow us to create forms of randomness with shaders. In particular, we will be using value noise, which looks like this:

If we take multiple outcomes of value noise, combine them at different scales, warp the result⁴ so the lines become wiggly, and then use 2 thresholds to separate the values into 3 colors, we can achieve a fairly realistic looking koi pattern!

All that's left to do is to place this noise map onto the fish. Unfortunately, we can't exactly take a cutout of this image and apply it to the fish every frame. One fish moving counterclockwise would have a different pattern than the same fish moving clockwise!

The solution to this is to use UVs. UVs are essentially extra coordinates (in addition to the vertex coordinates) that tell the shader which vertex to map to which part of the texture. This way, even if the vertex (and fish) moves up/down/left/right, it will still point to the same spot on the texture, and therefore output the same color.

The full shader code used to generate patterns is shown below, including the noise primitives:

const vec2  HASH_K   = vec2(123.34, 456.21);
const float HASH_ADD = 45.32;
const float FBM_LACUNARITY = 2.0;    // each octave doubles the frequency...
const float FBM_GAIN       = 0.5;    // ...and halves the amplitude
const float ASPECT = 3.0;            // body UV is wider than tall; keep noise cells square in world units
const float WARP_AMP  = 0.55;
const float WARP_BIAS = 0.5;         // centers the warp push on 0 so the pattern doesn't drift
const float WARP_FREQ = 1.2;
const float N1_FREQ   = 1.6;
const float N2_FREQ   = 2.3;
const float N2_OFFSET = 7.0;         // de-syncs n2 from n1 so accents don't sit on top of blobs
const float T1 = 0.5;
const float T2 = 0.58;

// Deterministic pseudo-random in [0,1) from a 2D point.
float n_hash(vec2 p) {
  p = fract(p * HASH_K);            // scramble into [0,1)^2
  p += dot(p, p + HASH_ADD);        // mix x and y so changing one shifts both
  return fract(p.x * p.y);
}

// Value noise: hash the integer lattice, bilinearly interpolate with a smoothstep.
float n_vnoise(vec2 p) {
  vec2 i = floor(p);                              // cell origin
  vec2 f = fract(p);                              // position within cell [0,1)
  vec2 u = f * f * (3.0 - 2.0 * f);               // smoothstep weights to remove diagonal seams
  return mix(mix(n_hash(i),              n_hash(i + vec2(1, 0)), u.x),
             mix(n_hash(i + vec2(0, 1)), n_hash(i + vec2(1, 1)), u.x), u.y);
}

// Fractal Brownian motion: sum value noise across `oct` octaves.
float n_fbm(vec2 p, int oct) {
  float v = 0.0;
  float a = FBM_GAIN;                  // amplitude of the next octave
  for (int k = 0; k < oct; k++) {
    v += a * n_vnoise(p);
    p *= FBM_LACUNARITY;               // zoom in for the next octave
    a *= FBM_GAIN;                     // ...fading it down each pass
  }
  return v;
}

// Analytic AA across a threshold of a noise field, using the field's screen-space derivative.
float n_edge(float n, float t) {
  float aa = fwidth(n);                // ~how much n changes between neighboring pixels
  return smoothstep(t - aa, t + aa, n);
}

And the combination of such primitives into the pattern:

#version 300 es
precision mediump float;

in vec2 v_uv;            // x -> body u, y -> body v, all in [0, 1]
uniform vec3 u_base;     // belly / background color
uniform vec3 u_mid;      // primary blob color
uniform vec3 u_accent;   // accent flecks
uniform vec2 u_seed;     // per-fish offset so each koi reads differently
out vec4 o;

#define FBM_OCT 3
#define WARP_OCT 2

void main() {
  vec2 uv = v_uv;
  vec2 st = vec2(uv.x * ASPECT, uv.y);                             // world-isotropic noise space
  float q = n_fbm(st * WARP_FREQ + u_seed, WARP_OCT);             // low-freq field used only as a warp source
  vec2  w = st + WARP_AMP * (q - WARP_BIAS);                       // domain-warped sample point
  float n1 = n_fbm(w * N1_FREQ + u_seed,                FBM_OCT); // main blob field
  float n2 = n_fbm(w * N2_FREQ + u_seed.yx + N2_OFFSET, FBM_OCT); // accent field, seeded differently so it doesn't overlap n1
  vec3 c = u_base;
  c = mix(c, u_mid,    n_edge(n1, T1));                           // paint blobs over base
  c = mix(c, u_accent, n_edge(n2, T2));                           // paint accents over that
  o = vec4(c, 1.0);
}

Our fish now actually look like koi fish!

Fish Behavior#

The fish behavior as a whole can be split up into several factors, each working together to dictate a fish's heading. Certain factors are weighted more than others, which allow fish to avoid a mouse cursor, while still wandering if they'd like. All in all, the factors are:

• Wandering (1x): another noise texture is sampled, which determines which direction the fish should go in, without any other factors.
• Containment (0-5x): as fish get closer to the edge, the factor pushing the fish away gets stronger. This keeps fish centered on the screen.
• Avoidance (0-7x): weight ramps up as fish get closer to the cursor, causing them to dart away if near danger.
• Seeking (6x): fish seek any treat within a 1024 pixel radius (they are hungry, feed them)!

The amount the fish can turn in one frame is capped, so the fish dont randomly bend in half. Finally, variations in speed are added depending on behavior in order to make the fish a little more lively and sentient-looking.

Lotuses, Lilypads#

As you might have guessed, the flowers and lilypads are procedurally generated as well! Lilypads are made using a series of triangles in a pie. Then, a shader is used to darken the outer edges, remove a wedge, and add some texture.

Lotuses are made using a series of triangle-ellipse love children. Vertex positions for an ellipes and triangle are generated. Then, the average is taken between them in order to generate a rounded triangle! Each of these rounded triangles are then rendered with a shader that brigntes the tip of the petal, just like actual lotuses!

These rounded triangles are then layered on each other in rings towards the center, ensuring that any gaps are covered.

Instancing#

One might notice that there are a lot of these petals on the screen at any particular moment. Each of these petals additionally has many, many vertices. The CPU must tell the GPU to render all of these vertices and triangles for every single petal in the scene! In these situations, we can use a technique called instancing in order to optimize performance!

Instancing works by giving the GPU a batch of identical meshes. Then, we pass in data to the GPU called instance data, which tells the GPU how to render each individual mesh. By passing all of the petals to the GPU like this, we can prevent the CPU from uploading the same data, over and over again.

This pattern sound familiar? The properties of our specific use case (rendering duplicate meshes over and over) led to a non-general optimization, yet again.

While we are here, we can apply instancing to the rest of the items in the scene, like fish fins, fish eyes, and lilypads. While we probably won't see drastic improvements, optimizing draw calls will still help future proof our project (just in case we'd like to render 10000 flowers).

Water Shaders#

Right now, our water is just one solid color. Its flat, and doesn't look quite like water at all. For this project, I wanted to render water with a "toon" shader, a type of shader that uses clean edges and distinct colors to give the water a cartoon feel.

Water shaders usually consist of 3 parts. First, the deeper the water is, the more light it should reflect. Fish at deeper levels should appear more blue. Second, water should foam up in shallow water and against objects in the water, as well as cause ripples. Third, water should bend the light as it penetrates the water's surface, causing the scene behind the surface to warp. To tackle this problem efficiently, we once again find optimizations for our use case!

Tinting#

Let's handle the first feature. Because our scene is 2D and perfectly top down, we can just tint the fish more blue the deeper it is. No need to add extra complexity, or render multiple water quads.

Foam#

The second case is what I'm least happy about. Traditionally, 3D water shaders use depth testing to identify areas near land: if the difference between the water height and the bank height is small, that means that something emerges from the water soon. Unfortunately for us, there is no depth to test in 2D.

As all of the objects that sit on the water should produce roughly circular ripples, we can simply hardcode the foam to be circular (allowing a wedge to be cut out, for lilypads). We render each ripple to two textures, one representing the intensity of the foam/reflection, and one representing the intensity of the displacement caused by the ripple.

Then, we reference these two textures while rendering the water, using the intensity texture to determine the color, and the displacement texture to determine the refraction offset (more on this later).

Refraction#

The third case is more interesting. By rendering the scene from the bottom up, we can notice that by the time we want to render the water layer, everything under the water has already been rendered. To refract the water, we can simply sample this already-rendered layer, displacing each pixel by a certain offset.

"How do we know how much to displace each pixel?" you might ask? Well, with noise of course! In this case, we use Fractal Brownian Motion, the same technique we used to generate koi patterns, shown below:

Remember the displacement mask that we mentioned in the previous section on ripples? We can sample from that texture in order to vary the intensity of the water refraction. See it in action below, where the refraction is stronger next to the ripples of the lilypad:

However, it turns out that calculating fractal brownian motion for every pixel, every frame, is still quite taxing on the GPU. Fortunately for us, we don't have to do that! Simply sliding a FBM texture (generated once) across the screen gives us a result we are happy with:

Yippee! Now, we have water shaders that make the scene look much more alive!

Efficient Stylized Shadows#

Despite all of this, our scene is still looking rather flat. That's because it is: there are no shadows at all! Lotuses and lilypads should cast shadows on the water, fish should cast shadows on other fish, and these shadows need to dance to reflect the motion of the water. Still so much to do!

Once again, we are going to use certain properties of our use case in order to come up with a solution that is efficient, but still provides the results that we want. We can see that we only have one single light in the scene: the sun. The sun (being the sun) is very far away, which means that shadows will remain the same size as the object that casts them, and shadows will always be cast in the same angle.

To determine whether a pixel is shadowed, the questions we need to answer are: "is there an object above me?" and "how far is that object from where I am?" It turns out one easy way to answer these questions, is yet again, with the help of another texture.

We start by rendering a heightmap, which effectively sits at the bottom of our pond floor. In this texture, a pixel represents the height of the tallest object that would cast a shadow at that pixel. Once we actually render an object, we check the heightmap to see if the current pixel would obstruct a shadow, and if so, at what height the shadow was cast from.

If that value is greater than our current height value, then we darken the current pixel relative to the distance between them. To make these shadows seem less like cutouts, we also add displacement to the shadow if it passes through the water surface. Voila! Shadows that make our scene much less flat.

Final Optimizations#

I won't dive too deep, but several optimizations can still be made here. These include:

• Decreasing the render resolution if a device can not render the scene at 60fps.
• Reducing the number of allocations and inlining some function calls to reduce lag spikes from garbage collection.

In general, you should always profile your code when deciding what to optimize next. Eliminating one memory allocation doesn't do anything if you are rendering a terribly optimized shader, for example. You can press p to see more information about how long each part of the scene takes to render!⁵

Conclusion#

Ultimately, the takeaway from this blog (besides you learning about the tech behind the koi pond) is to realize that pretty much everything in software engineering has a trade off. This could be between memory complexity vs time complexity in some program, or it could be between code readability vs performance in the code you write. In our case, we needed to utilize the less-general and harder-to-use features of WebGL to control and optimize our code. We also sacrificed realism⁶ for efficiency and a "toon" style.

Regardless, this is my 6th personal website, and it is one that I'm finally really happy with. Hopefully that stays the case!