Shader tints model in red with no obvious reason

[Lunarovich]

Hello! I have this minimalistic quad model:

 

local model = Model:Create()
local surface = model:AddSurface()

surface:AddVertex(-0.5,-0.5,0, 0,0,-1, 0,0)
surface:AddVertex(0.5,-0.5,0, 0,0,-1, 1,0)
surface:AddVertex(0.5,0.5,0, 0,0,-1, 1,1)
surface:AddVertex(-0.5,0.5,0, 0,0,-1, 0,1)
surface:AddTriangle(2,1,0)
surface:AddTriangle(0,3,2)

surface:UpdateAABB()
model:UpdateAABB(Entity.LocalAABB)
model:UpdateAABB(Entity.GlobalAABB)

 

In order to shade it, I create a simple material and give it my custom minimalistic shader. Here is the vertex shader

 

#version 400
#define MAX_INSTANCES 256

//Uniforms
uniform instancematrices { mat4 matrix[MAX_INSTANCES];} entity;
uniform mat4 projectioncameramatrix;

//Attributes
in vec3 vertex_position;

void main()
{
mat4 entitymatrix = entity.matrix[gl_InstanceID];
mat4 entitymatrix_ = entitymatrix;
entitymatrix_[0][3]=0.0;
entitymatrix_[1][3]=0.0;
entitymatrix_[2][3]=0.0;
entitymatrix_[3][3]=1.0;

vec4 modelvertexposition = entitymatrix_ * vec4(vertex_position, 1.0);

gl_Position = projectioncameramatrix * modelvertexposition;
}

 

And here is the fragment shader

#version 400

// Outputs
out vec4 fragData0;

void main()
{
fragData0 = vec4(0.0, 0.0, 0.0, 1.0);
}

 

Now, you should notice that I do not give any special color to the model, material or vertices, and that I set fragData0 to black, so normally, it should be black (and since the background is black also, it should be invisible in this particular case). However, it is red:

 

 

If I change the relevant line of code to, e.g. fragData0 = vec4(0.0, 1.0, 0.0, 1.0); I get a greenish mix of green and red:

 

 

Or with fragData0 = vec4(0.0, 0.0, 1.0, 1.0); I get purple, as expected...

 

 

How can I get rid of this basic red tint? Why is it there in the first place?

 

Thank you!

[macklebee]

The shader is not writing to fragData1 which will cause the red tint. Refer to the model shaders for details.

[Lunarovich]

Thank you! Finally, the problem was in fragData2. Here is a version of the fragment shader that gives correct results:

 

#version 400

// Outputs
out vec4 fragData0;
out vec4 fragData2;

void main()
{
   fragData0 = vec4(0.0, 1.0, 1.0, 1.0);
   fragData2 = vec4(0.0,0.0,0.0,0.0);
}

 

According to this post, fragData1 stands for normals and specularity, and fragData2 encodes emissivity and something called material flags. As far as I can see it, my material was, for some reason unknown to me, emitting the red "light". I've turned it off by setting every value of fragData2 to 0.0.

 

I've made some tests with fragData2 and it does affect the final color of the model. What I see as a design flaw is this: if you do not specify some color value and material needs it for some reason, the default value should be set to no particular value (black in our case) or to some neutral value (white, in the case of tinting, for example). The default value should not be some idiosyncratic color like red.

[Rastar]

Leadwerks uses a little trick to render selected items differently (this is relevant for the editor). If you look e.g. into the fragment shader of default.shader

 

if (ex_selectionstate>0.0) materialflags += 2;

 

so if an entity is selected, the corresponding material flag will be set. Now, in the lighting shaders (e.g. Lighting/directionallight.shader) you'll find the lines

 

 //Blend with red if selected
 if ((2 & materialflags)!=0)
 {
  sampleoutput = (sampleoutput + vec4(1.0,0.0,0.0,0.0))/2.0;
 }

 

so red will be added to the output color.

[Lunarovich]

Thanks for this clarification. Then again, models shouldn't be selected by default... And their selection state should not be reflected in any way during the runtime...

 

Anyway, in the default shader, for example, in the vertex part, I see these lines:

 

   ex_selectionstate = 0.0;
   if (vColor.a<-5.0)
   {
       vColor.a += 10.0;
       ex_selectionstate = 1.0;
   }

 

As far as I can see, the alpha value of the entitymatrix[3][3] (I refer to the entity.matrix[gl_InstanceID]) encodes the selection state of the object. Am I right? I mean, vColor.a is fed with value via

 

vColor=vec4(entitymatrix[0][3],entitymatrix[1][3],entitymatrix[2][3],entitymatrix[3][3]);

 

So, here comes the puzzling part. If this value is less than -5, than we add 10 to this value. Why do we want to change it? I see that ex_selection "boolean" gets the value of 1, which I understand (or I think I understand).

 

Anyway, in the fragment part of the shader, one can read this:

 

   int materialflags=1;
   if (ex_selectionstate>0.0) materialflags += 2;
   fragData2 = vec4(0.0,0.0,0.0,materialflags/255.0);

 

According to this post, material flag 1 stands for ligthing enabled and 2 stands for decals enabled (reserved; btw, why reserved? what does it mean?). So, if we add 2, material flag will be 3. What does it mean for materialflags to be 3? And finally, if we divide materialflags by 255, even when it's 3, we'll have a value so small that it is barely (if at all) visible to the human eye. What's the point in all this?

[shadmar]

The flag is used when you select something in the editor, it then gets colored red.

So you really dont need it outside the editor.

[Lunarovich]

Thanks. That makes things a bit simpler.

 

Anyway, I'm still curious about all those lines of code, so an explanation or hint would be appreciated. I'm currently compiling my own notes in regards to how one uses shaders in LE. Maybe I'll put them on blog here, as a newbie guide someday.

[Rastar]

This is really a bit mask. You start with a value of 1 (2^0), which in binary is 00000001. If you add 2 (2^1), this means the second bit flag from the right will be set as well (00000011). So the actual number 3 doesn't have a special meaning, just which bits are true or false.

 

Now, the value is stored as a number between 0.0 and 1.0 in fragData2. So you have to divide the value by 255 (equivalent to 11111111).

[Lunarovich]

Thanks! That was quite helpful. I suppose that somewhere down the pipeline we convert materialflags back to bit mask by multiplying it with 255.