Masking is pretty easy and useful for adding some variety to your shaders. Masks are usually just some greyscale image that is turned into a float. In shaders floats have a very broad utility because you can add them just about anywhere. Super useful. And they can be used to blend textures or define areas in your material.
Adding a mask in code is the same as with any texture. The greyscale quality allows you to use a single channel of that greyscale as a representative of all the channels. For instance the red channel. In code it looks something like:
float mask = texture(texture5,ex_texcoords0).r;
In blending two textures you need the two textures you want to blend and a third mask texture to blend them by. Once you've established them in the shader you simply do this in the output line:
fragData0 = mix(outcolor,outcolor2,mask);
That's it. The mix function is pretty much the whole trick. But there are other things you can do. One variation of texture blending is by using an alpha mask in what I believe is referred to as "texture splatting."
If you set your texture with an alpha channel in your image manipulation program and then in the texture editor set the texture compression to something other than DXT1 you can get an alpha blended texture with a smooth falloff (the alpha wont affect the whole object. Just the texture). Make sure when you're doing this you don't use .jpg image types as jpegs don't carry the alpha channel across. I use .png
In code that looks like this:
float mask = texture(texture5,ex_texcoords0).a;
Note I used ".a" at the end to signify the alpha channel. Here's a nice moldy example of that:
Below is your standard two-color blend. It gets a little tricky adding the normals. I think in this one I just did the entire normals section twice but it just comes down to mixing the normals by the mask so I believe you could get away with something like this:
vec3 normal = ex_normal;
normal = normalize(texture(texture1,ex_texcoords0).xyz * 2.0 - 1.0);
vec3 normal2 = normalize(texture(texture6,ex_texcoords0).xyz * 2.0 - 1.0);
normal = mix(normal,normal2,mask);
normal = ex_tangent * normal.x + ex_binormal * normal.y + ex_normal * normal.z;
normal = normalize(normal);
This three-color mask actually uses an rgb mask rather than a greyscale. The shader looks exhausting but really I'm just doing the same thing over again three times - once for each color. You may notice on the right there that I ran out of spaces so I just use a generic method for the specular. It's kind of cool you can just change where the colors show by changing the mask. Make sure in making masks for this that you use really really blue, really really red, and really really green otherwise your textures will blend into areas you don't want them:
Here I used a cubemap to get some reflective little puddles in the mix and mixed the specular and normals by the mask without changing the diffuse:
This one doesn't even use a mask but I think it's cool. It uses "normal.y" as a means of mixing textures. So if you want to add snow to the top of a rock, for instance, you can use this shader to do so. The blend settings are adjustable:
Open your two-color blend shader and find the "float mask =.." line around line 54 and change it to " float mask = 1.0-normal.y; " ("1.0 - something " just inverts the thing), compile (press the "play" button on the Script Editor), and watch what happens. Or change that line to " float mask = gl_FragCoord.w + 0.35; " and check that out. Once you figure out your float you can pretty much mix your textures by anything.