August 15, 2016

Water for Games - Cheap, Simple, Effective

71% of the Earth's surface is water. Yet, of all the materials I've ever made, water seems to be the most difficult to get right. Even in UDK with planar reflections and Phong shading, water was still incredibly complex. Once you crack the code to good water, your game can turn from a nice scene to a gorgeous, luscious environment. For the sake of this blog post, I will avoid delving into vertex displacement and dive right into translucent rendering. This kind of water is good for a calm ocean, a shimmering lake, the famous Carribbean seas, inter-coastal waterways, and pool water, with more emphasis on translucency and reflective effects. Raging oceans and rivers will be very different.




I first really got excited about realtime water with Super Mario Sunshine. For years I crowned its water as a technical and artistic achievement of the highest order. Keep in mind games like Goldeneye 007 released just 4 years prior to this game, and this was before we got a chance to experience the water from Half Life 2, Assassin's Creed, Uncharted, and Crysis. Before all these games, there was Super Mario Sunshine. The water was just so blue and seemed to look so beautiful without ever being inappropriate. But as it turns out, the water in Sunshine was, for the most part, a hoax. The water surface itself was just a texture that bobbed up and down with the waves. At close distance the water was completely translucent, while the "shimmering" texture would be emphasized further away. The water itself was clear: objects underneath the water were vertex painted to a clear blue color giving the water that strong Caribbean blue saturation. Sunshine used custom mipmaps to push the shimmering effect away from the player, but we can use other means to simulate this.

In real life, water reflects, refracts, and scatters light. You can make the shader much easier to render by mimicking the results instead of the effects. The combination of reflections and refractions looks like turbulence that bends around the bulges in the water's surface. Specularity can help to define actual reflections on the surface. The scattering can be simplified to a depth-based color.

Pros to my water technique (inspired by a combination of Super Mario Sunshine and Super Mario Galaxy's techniques):
  • Cheap! 45 instructions for texture highlights, 72 for Phong specular, 74 for GGX specular, 76 for texture+GGX specular.
  • Fluid! Looks like flowing, fluid water.
  • Translucent! You can see objects underneath. And those objects have proper depth.
  • Distance opacity! The depth method inherently makes the water more "reflective" in the distance and more translucent up close.
  • Tilable, but doesn't look it! 2048 and different tiling factors between the distortion asset and texture/normal asset reduces tiling in the long range.
  • Memory and texture sampler efficient! While the 2048 texture can seem like a lot, you can hide this in a channel of a compressed 2k mask, effectively getting this endless ocean for less than 1MB of texture data. And if you use 1k maps, you can cut that size by 1/4. And the distortion normal map only needs to be 256x256 pixels. Two textures for beautiful water!
  • Flexible. Since the lighting is handled using forward-rendering techniques, you can get benefits out of this system that you can't with UE4's deferred renderer, like changing the color of the specularity, coloring by sphere maps, iridescent effects.
  • Supports day/night cycle. By updating the sun's rotation in blueprints, the specularity on the water can change according to the sun's position.
  • Supports color gradient by depth. Expanding the depth fade and lerping the result to different colors, you can make deeper waters a much darker blue than shallow areas.
Limitations of my water technique:
  • No shadows. While you can multiply the specularity by a custom texture map or through vertex painting, there is no real process to get shadows baked on here.
  • Only supports one light. The sunlight. This method will not work for reflections from lights on a pier. But it can provide general water shading to support such methods.
  • Some manual effort required to provide light direction. 




All you need to create this effect is a simple distortion texture (a tiling bubbly normal map), a shimmery, circular noise grayscale texture, and that's it! For a more advanced version with GGX specular highlights from the sun, you'll also need a normal map of that circular noise grayscale texture.




This material only costs 46 pixel shader instructions and 45 vertex shader instructions despite being a translucent material with depth fading. If you have a sand texture underneath, this water will be cheaper to render than the sky. Making the material unlit and removing vertex fog will eliminate a ton of instructions.

In the picture above, the Roughness GGX Specular is not even used, so the distortion is actually doing most of the heavy lifting for this effect. Pan the bubbly normals, mask RG, and scale down by 0.05 to 0.1, or however strong you wish the distortion to be. Add that to the UV coordinates of your water texture (tile it as many times as you need). If the texture is imported as an uncompressed linear grayscale, multiplying it by itself will give you a cheap gamma-corrected version to add on. Multiply it by the brightness you wish the highlights to be, and add it to your ocean color. A deep saturated cerulean blue with some extra intensity works great. The depth fade in opacity will make your water more translucent with objects closer to the surface and more colored when objects beneath the water are further away. For an ocean and most water surfaces in general, this means you will get a more opaque appearance in the distance.

To get the right texture you need, start with a 256x256 pixel grayscale noise. Enlarge this to 2048x2048. Use the gradient map tool to make white circular bands according to the noise texture. Two bands through the gradient map adjustment is enough to get a good effect. Convert this "heightmap" to a normal map, and you can use the normal map instead (or both).




The GGX specular method calculates a specular highlight using the same principled GGX specular code that UE4 uses for calculating specular highlights for stationary and dynamic lights. The HLSL GGX code in the custom node is:

float a = Roughness * Roughness;
float a2 = a * a;
float d = ( NoH * a2 - NoH ) * NoH + 1;
return a2 / ( PI*d*d );

And the two inputs are Roughness and NoH, where NoH is the Blinn model. I added a Lambertian diffuse and multiplied it by itself to get a more focused, gamma-corrected result. The light vector is pulled from a Blueprint actor that is placed in the level and automatically grabs the directional light/sunlight's rotation.




This code will then feed to a Material Parameter Collection that communicates with the material to set up a specular highlight on the water.




This version of the water shader replaces the texture highlights with the GGX specular highlights calculated from the normals, so there is no texture 74 pixel shader instructions, 2 textures.




And this version adds both the texture and the normal specular for highlight rendering. The texture is not multiplied, only added. 76 instructions, 3 textures.


Who Wants to Be a Millionaire: Breakdown of Sound Design

I might be making a quiz game soon for a client, and I think I can learn from the design decisions of what has to be the greatest quiz game ever made, bar none: Who Wants to Be a Millionaire. More specifically, the sound and music. The sound doesn't just get more ominous as the game progresses, the sound design is actually quite complicated with many surprising twists along the way.

$100, $200, $300, $500, $1,000 Questions: https://www.youtube.com/watch?v=P4utAJ-JraE

At the beginning, there are two big "let's play" tracks for the start of the game at $100 and the continuation before $200. This is a fanfare that hones in the game at the beginning. The game as a whole is designed to be intimidating and make the player feel very uncomfortable. This makes winning high jackpots feel very rewarding. At the game's beginning, sound effects for wins are very unobtrusive and lightweight. The music is lighter, and bouncier, and more sensitive to wins. There are no sounds just for answering or playing because the assumption is you should be able to pass each question without trouble. It helps push players up and out of this section to move on to harder questions and bigger risks.

The surprise here is, why bother wasting time on questions you know people can answer? Shouldn't the whole game be intimidating? This section is included in the game to give it a fast start and a more dynamic introduction. You pull people in with these questions. You challenge them later on. Again, the real surprise here is that the soundtrack is actually more repetitive, less solemn, and less epic than the previous phase. You'd think the sound should get lonelier as you approach the top, but it doesn't. I believe this is, ironically, intended to push players closer to the $1,000,000 question faster by means of the anxious repetition, and from watching the show we know this is where many people screw up very quickly and drop back down to $32,000. The drama of the risk involved was soaked for all it was worth in the last phase, but now that the jackpot is so great, it would be much more interesting just to see someone just make it to that last question, and the faster, more repetitive music divides to accomplish both: it pushes the brightest to the top quicker while the lesser freefall sooner.



After the $1,000 mark, the music undergoes a dramatic change, getting much quieter, more ominous, and just selecting an answer now has a sound effect. Surprisingly, and I didn't realize this until after researching, every question's sound ascends keys in minor scale from $2,000 all the way up to $32,000. This is true for both the question's music and the final answer sound effect. You'd think the music should descend the scale, but the purpose of having higher sound as you progress is to indicate the rising jackpot, and the rising risks. The majority of the game is spent in this segment, squeezing the drama for all it's worth.



The $64,000 question marks another significant change as we go back to the bottom of the scale, this time with a different, more repetitive "heartbeat" soundtrack, interspersed with the ambient sitar for more atmospheric connotations. However, because the song is completely different, the change in scale is not as noticeable until the final answer sound effect at the end of the $64,000 mark. It is the same sound, same scale, same key as the $2,000 answer. Compared to the higher tone of the previous $32,000 question, this moment doesn't just mark the final phase as beginning, but as having already begun, and it catches everyone by surprise.


The final question is a doozy. It is the most repetitive, the most ominous, the most solemn, and of all the tracks heard by far it is definitely the darkest track of anything I've ever known. It's just a 3 second looping beat and a droning pad that never stops. It's easy to see why this choice was made as no other soundtrack could possibly be more conspicuous than a repetitive 3 second loop, but it flies in the face of game shows that try to make their larger prizes more like bonus rounds and fun, happy, crazy dances. This is not. This is the final question. There are no more questions after this. If you lose, you risk $468,000. If you win, you will be a millionaire.

Overall, the progression follows the show's intimidating and daunting experience, using key and soundtrack changes to push players through the easy questions, dramatize the difficult questions, and force them to fall quickly or climb to the top. The feedback is so subliminal, but very much intended by design as though it was a deliberate story. The complexity of Millionaire's sound design is something that many developers should learn from regarding how to use sound to set the drama of an interactive game into a story.

August 1, 2016

Rolling Displacement: Cheap AND Good-Looking Cloth for Console and PC

88 instructions - The number of instructions necessary to render beautiful silk cloth
75 instructions - Vertex shader cost of 3D cloth displacement

I have spent years trying to get APEX cloth to work on characters without needing NVIDIA's APEX integration with Maya or 3DS Max, but I cannot figure out how to authorize it, nor have I seen anyone else do it. In my open world project, Jake, the main character was always butt-naked. He was intended to have a red flowing cape. My folk's chargin with having a naked character and insistence that he wear something made me decide to try a different tactic: cloth displacement and normal calculation from textures. And surprisingly, the results from this method look even better than low-density APEX simulations.



The overall gist of this method is to offset sine and cosine waves flowing down the cloth by textures to flow in a unique pattern, transform from local space to world space and combine them to form the resulting vertex displacement, and using the sine data to drive a normal calculation. To the best of my knowledge, nobody has come up with a displacement/normal driven technique like this before. I'm calling this method "Rolling Cloth" due to the way the sine waves roll with the curvature of bends in the cloth rather than across them.

The benefits of this method are:
  • Extraordinarily cheap and detailed cloth displacement (great for console, mobile, and PC, assuming the API can handle texture displacement)
  • Rolling normals calculation looks great on low-poly cloth, even with no displacement at all
  • Highly scaleable to intended hardware
  • Normal calculation is fairly accurate
  • Artistically driven by textures for great flexibility in final results
  • Looks great with smaller textures (64-128 pixels)
  • Easily supports tiling cloth texture normals
The limitations of this method are:
  • Requires API to be capable of running textures to vertex displacement
  • Wind is manually driven by attributes in local space, not automatic in world space
    • Oddly enough, this is why the shader does not support triangle explosion through tessellation, because transforming from local to world space is incompatible with tessellation
  • The cloth is NOT simulated, so it will pass through world objects and characters
  • Movement rolls with bends in the fabric instead of across them

The whole material setup and finished result. Not too complicated, but looks great!
UVs on my cloth are set up so I can use a simple texture coordinate gradient as a mask and flow direction. In the mask, black areas will be swayed before white areas. The mask may require some fiddling to get right. For the trigonometry waves, I suggest the Face and Vertical waves are Sine and Cosine of the same period so you get a nice self-preserving circular motion. The negative of the first sine wave will just yield an unnatural linear displacement diagonally. The Lateral wave should be different than the first two swaying back and forth. If you want to prevent the cloth from looping, you can adjust the lateral motion or add more waves here at different periods.
The strengths of the Face/Lateral/Vertical transforms are purely visual. Eyeball it. If it looks right, it is right. The normals calculation, though, works the best when divided by the number of  transforms, and when it's scaled to their intensity. You can replace the 0,0,1 in the normals generator with a tiling normal map and it will work just as well. That "Normal of Mask" is actually a normal map of the grayscale mask. It's important the mask and normals are smooth, so don't be afraid to use a gaussian blur or low-res textures for them.
Multiply each wave times the mask, then add them together for the normals calculation. For the transform, multiply each transform by its corresponding Wave*Mask result, then simply add the results together. For this cloth, I wanted a silky cloth, so I used a 0.7 semi-metallic level and 0.6 roughness. You can mess with these numbers for different types of silk. Subsurface looks the best when I multiply the color by 0.2. This material is Two-sided, and using the Two-sided shading model. Cloth shading can yield nice cloth results as well. You can even use an opacity mask to show holes through the cloth.