How do I use stencil?

Questions about the LÖVE API, installing LÖVE and other support related questions go here.
Forum rules
Before you make a thread asking for help, read this.
Post Reply
User avatar
Vimm
Party member
Posts: 113
Joined: Wed Mar 16, 2016 8:14 pm

How do I use stencil?

Post by Vimm » Thu Dec 01, 2016 8:33 am

I'm trying to use stencils to make it so when a certain object spawns in my game, it can only be drawn on tiles from Tiled, and not on the background. I've looked at the wiki but trying to implement it gives me a headache :/

The wiki shows it all in one file, main.lua, but my stuff is spread out, all the tiled stuff is in the map_loader, and the object i wanna draw is in its own file as well, my major issue is figuring out what goes where, so if I could get some help that'd be awesome :)

User avatar
pgimeno
Party member
Posts: 1989
Joined: Sun Oct 18, 2015 2:58 pm

Re: How do I use stencil?

Post by pgimeno » Thu Dec 01, 2016 5:54 pm

Only you know the organization of your game, but the functionality is as follows.

You can see the stencil as an invisible mask, similar to an alpha layer, but only yes/no. When the stencil is active, a pixel is either drawn or not drawn, even if the values of the mask can be in 0-255; there's no opacity. You control the mask value threshold for a pixel to be drawn.

You draw to that mask (the stencil) with regular draw functions. What gets drawn in the stencil and what gets drawn in the screen, is decided by whether you're drawing inside the stencil function or outside. For example:

Code: Select all

  love.graphics.stencil(function ()
                          love.graphics.rectangle("fill", 10, 10, 20, 20)
                        end,
                        "replace", 1, false)
  love.graphics.setStencilTest("greater", 0)
  love.graphics.circle("fill", 30, 30, 20)
The "replace" mode means that the pixels that are drawn will all be set in the mask to the value we supply (1 in this case). The colour and alpha you choose doesn't matter; everything you draw, be it a circle, a rectangle, lines, or whatever, will have the value 1. Since we're setting 'keepvalues' to false, the mask will be cleared before drawing anything to it (similar to a love.graphics.clear of the mask).

The rectangle will be drawn to the stencil; the circle will be drawn to the screen.

Note that once the stencil is drawn to, you have to specify the threshold value for a pixel to be drawable. That's what setStencilTest does. In this case, we're saying that any value greater than zero is drawable. We have drawn the rectangle with the value 1, so all the pixels in the rectangle will be drawable.

Now, as for how to adapt it to your game, exactly how is up to you, but it boils down to this. You would draw the tiles to the stencil as well as to the screen. Assuming you don't need the stencil for other uses, you can just draw the tiles to the screen and to the stencil at the same point of the code. For example:

Code: Select all

local function myStencilFunction()
  drawTiles()
end

...
    drawTiles()
    love.graphics.stencil(myStencilFunction) -- default values for the rest are OK
That prepares the stencil. Then before drawing the objects to spawn, you do:

Code: Select all

  love.graphics.setStencilTest("greater", 0)
and when you finish drawing the objects to spawn, you do:

Code: Select all

  love.graphics.setStencilTest()
That should be all.

User avatar
Vimm
Party member
Posts: 113
Joined: Wed Mar 16, 2016 8:14 pm

Re: How do I use stencil?

Post by Vimm » Thu Dec 01, 2016 6:59 pm

pgimeno wrote:Only you know the organization of your game, but the functionality is as follows.

You can see the stencil as an invisible mask, similar to an alpha layer, but only yes/no. When the stencil is active, a pixel is either drawn or not drawn, even if the values of the mask can be in 0-255; there's no opacity. You control the mask value threshold for a pixel to be drawn.

You draw to that mask (the stencil) with regular draw functions. What gets drawn in the stencil and what gets drawn in the screen, is decided by whether you're drawing inside the stencil function or outside. For example:

Code: Select all

  love.graphics.stencil(function ()
                          love.graphics.rectangle("fill", 10, 10, 20, 20)
                        end,
                        "replace", 1, false)
  love.graphics.setStencilTest("greater", 0)
  love.graphics.circle("fill", 30, 30, 20)
The "replace" mode means that the pixels that are drawn will all be set in the mask to the value we supply (1 in this case). The colour and alpha you choose doesn't matter; everything you draw, be it a circle, a rectangle, lines, or whatever, will have the value 1. Since we're setting 'keepvalues' to false, the mask will be cleared before drawing anything to it (similar to a love.graphics.clear of the mask).

The rectangle will be drawn to the stencil; the circle will be drawn to the screen.

Note that once the stencil is drawn to, you have to specify the threshold value for a pixel to be drawable. That's what setStencilTest does. In this case, we're saying that any value greater than zero is drawable. We have drawn the rectangle with the value 1, so all the pixels in the rectangle will be drawable.

Now, as for how to adapt it to your game, exactly how is up to you, but it boils down to this. You would draw the tiles to the stencil as well as to the screen. Assuming you don't need the stencil for other uses, you can just draw the tiles to the screen and to the stencil at the same point of the code. For example:

Code: Select all

local function myStencilFunction()
  drawTiles()
end

...
    drawTiles()
    love.graphics.stencil(myStencilFunction) -- default values for the rest are OK
That prepares the stencil. Then before drawing the objects to spawn, you do:

Code: Select all

  love.graphics.setStencilTest("greater", 0)
and when you finish drawing the objects to spawn, you do:

Code: Select all

  love.graphics.setStencilTest()
That should be all.

So if I understand correctly, in my case I'd be putting the DRAW_MAP_LOADER() stuff in the stencil function?
https://imgur.com/tXwdGgH
Sorry, on my phone right now haha

User avatar
pgimeno
Party member
Posts: 1989
Joined: Sun Oct 18, 2015 2:58 pm

Re: How do I use stencil?

Post by pgimeno » Thu Dec 01, 2016 7:39 pm

Can't say for sure without looking at your full code, but it seems so. If you want the map to act as a mask for your spawning objects, you'll have to draw the map to the mask (=the stencil).

Edit: Maybe you can just do: love.graphics.stencil(DRAW_MAP_LOADER)
(Note the parentheses are missing, because you're passing the function, not executing it and passing the result)

User avatar
Vimm
Party member
Posts: 113
Joined: Wed Mar 16, 2016 8:14 pm

Re: How do I use stencil?

Post by Vimm » Fri Dec 02, 2016 12:42 am

pgimeno wrote:Can't say for sure without looking at your full code, but it seems so. If you want the map to act as a mask for your spawning objects, you'll have to draw the map to the mask (=the stencil).

Edit: Maybe you can just do: love.graphics.stencil(DRAW_MAP_LOADER)
(Note the parentheses are missing, because you're passing the function, not executing it and passing the result)
I'll try that when I get home, I feel like this is one of those things that once I figure it out it'll be obvious, but until then I'm so lost xD

User avatar
Vimm
Party member
Posts: 113
Joined: Wed Mar 16, 2016 8:14 pm

Re: How do I use stencil?

Post by Vimm » Fri Dec 02, 2016 5:53 am

Vimm wrote:
pgimeno wrote:Can't say for sure without looking at your full code, but it seems so. If you want the map to act as a mask for your spawning objects, you'll have to draw the map to the mask (=the stencil).

Edit: Maybe you can just do: love.graphics.stencil(DRAW_MAP_LOADER)
(Note the parentheses are missing, because you're passing the function, not executing it and passing the result)
I'll try that when I get home, I feel like this is one of those things that once I figure it out it'll be obvious, but until then I'm so lost xD
Just doing love.graphics.stencil(DRAW_MAP_LOADER) doesnt crash the game, so thats good, but nothing also changes lol. so either it isn't working, or theres something else i need to add to make it work. I assume i need to add something to my blood_splatter object to make it only draw in the stencil mask, but i dont know what or how haha (I'm bad)

User avatar
pgimeno
Party member
Posts: 1989
Joined: Sun Oct 18, 2015 2:58 pm

Re: How do I use stencil?

Post by pgimeno » Fri Dec 02, 2016 8:13 am

After preparing the stencil by drawing on it, you have to activate it. Read my explanation again.

User avatar
Vimm
Party member
Posts: 113
Joined: Wed Mar 16, 2016 8:14 pm

Re: How do I use stencil?

Post by Vimm » Mon Dec 05, 2016 6:11 pm

pgimeno wrote:After preparing the stencil by drawing on it, you have to activate it. Read my explanation again.

I feel like this should be working, but the DRAW_BLOOD isnt being called at all, or at least not drawn.

Code: Select all

local function my_stencil_function()
	DRAW_MAP_LOADER()
end

function love.draw()
	camera:set()

	love.graphics.stencil(my_stencil_function, "replace", 1)
	
	love.graphics.setStencilTest("greater", 0)
	
	DRAW_BLOOD()

	love.graphics.setStencilTest()

	DRAW_BLOOD_PARTICLE()
	DRAW_PLAYER()
	DRAW_BLOOD_PROJECTILE()
	DRAW_SPIKE()
	camera:unset()
end

User avatar
pgimeno
Party member
Posts: 1989
Joined: Sun Oct 18, 2015 2:58 pm

Re: How do I use stencil?

Post by pgimeno » Mon Dec 05, 2016 6:25 pm

Looks like it should be working, yes (apart from the fact that you don't draw the map to the screen, only to the stencil, but that shouldn't prevent the blood from being drawn). Hard to help without a test case, sorry.

User avatar
Vimm
Party member
Posts: 113
Joined: Wed Mar 16, 2016 8:14 pm

Re: How do I use stencil?

Post by Vimm » Mon Dec 05, 2016 6:40 pm

pgimeno wrote:Looks like it should be working, yes (apart from the fact that you don't draw the map to the screen, only to the stencil, but that shouldn't prevent the blood from being drawn). Hard to help without a test case, sorry.
yeah well the map is invisible in game anyway so that part should matter haha.

Post Reply

Who is online

Users browsing this forum: Sky_Render and 16 guests