Canvas and shader coordinate weirdness

[nameless tee]

Hello everyone,

I was trying to (ab-)use Löve2D for 3D lately and ran into a weird inconsistency between rendering to a canvas and to the window. Now I'm unsure whether it's a bug or a feature.

When drawing to the window the normal way, the Y-axis get's flipped. This makes sense since Love2D is using the top left corner as origin, but it might be worth of a warning in the wiki that it's a left-hand coordinate system and that it affects culling.

When using a shader that doesn't use the transformation matrix, the Y-axis does not get flipped. I guess this is because the flipping usually takes place in the transform_projection. This might be worth a note on the love.graphics.newShader page, since it's not obvious that it should be this way. Since "position" is only a function that is called by the hidden part of the shader users might think that the flipping might happen somewhere else.

The weird thing is that, when rendering to a canvas, the flipping apparently does happen somewhere else. And since it also affects culling it must happen during rendering to the canvas and not just while drawing the canvas. It's very surprising and send me down the rabbit hole of questioning the projection matrix I was using when I stumbled into this. Because of the culling problem I first thought that Z got flipped or something like that before I figured out that flipping Y fixes it.

So is this a feature or a bug? Where does the Y-axis get flipped? Would it be OK to add the suggested warnings/notes to the wiki?

Attached is a minimal-ish example and a table of observation on it. Unfortunately "flipped" in the table means displayed the none-löve-ish way while in the rest of this post it means the non-opengl-ish way.

Code: Select all

|------|--------|--------|-|---------|--------|
| cull | canvas | shader | | flipped | culled |
|------|--------|--------|-|---------|--------|
| none | o      | o      | | o       | o      |
| none | o      | X      | | X       | o      |
| none | X      | o      | | o       | o      |
| none | X      | X      | | o       | o      |
| back | o      | o      | | -       | X      |
| back | o      | X      | | X       | o      |
| back | X      | o      | | -       | X      |
| back | X      | X      | | -       | X      |
| front| o      | o      | | o       | o      |
| front| o      | X      | | -       | X      |
| front| X      | o      | | o       | o      |
| front| X      | X      | | o       | o      |
|------|--------|--------|-|---------|--------|

[nameless tee]

New information on the subject: I just did some digging in the source (which is probably the tidiest and most readable open-source project I've seen yet) and found that when rendering to a canvas, Löve does stop flipping the Y-axis but starts reversing the vertex winding instead. When using a shader that ignores the projection matrix, this messes up the culling. And when displaying it in the window, the canvas gets drawn upside down.

Relevant excerpt from Graphics.cpp:

Code: Select all

	if (iswindow)
	{
		gl.bindFramebuffer(OpenGL::FRAMEBUFFER_ALL, gl.getDefaultFBO());

		// The projection matrix is flipped compared to rendering to a canvas, due
		// to OpenGL considering (0,0) bottom-left instead of top-left.
		projectionMatrix = Matrix4::ortho(0.0, (float) w, (float) h, 0.0, -10.0f, 10.0f);
	}
	else
	{
		bindCachedFBO(rts);

		projectionMatrix = Matrix4::ortho(0.0, (float) w, 0.0, (float) h, -10.0f, 10.0f);

		// Flip front face winding when rendering to a canvas, since our
		// projection matrix is flipped.
		vertexwinding = vertexwinding == vertex::WINDING_CW ? vertex::WINDING_CCW : vertex::WINDING_CW;
	}

	glFrontFace(vertexwinding == vertex::WINDING_CW ? GL_CW : GL_CCW);

Since I don't see any nice way how this could be done otherwise without introducing inconsistencies somewhere else, I guess this is intended behaviour. There does not seem to be any mention of this on the graphics.newShader page, so everything that's left of my original post is a suggestion to add this in the wiki.

[pgimeno]

My best guess is that rendering to a canvas does not flip, because otherwise, when rendering the canvas to the screen it would be flipped twice. Since flipping inverts the sign of the cross product, the winding needs to be changed in order for the polygons to be drawn at all when drawing to the canvas with backface culling active.