Stencil Help

Questions about the LÖVE API, installing LÖVE and other support related questions go here.
Forum rules
Before you make a thread asking for help, read this.
Post Reply
User avatar
Alp
Prole
Posts: 7
Joined: Wed Jul 10, 2019 2:48 pm

Stencil Help

Post by Alp » Sat Aug 17, 2019 10:56 pm

I'm trying to create a masking effect for a character select screen, and I can't seem to make heads or tails of how stencils work.

It really doesn't help, that the only examples I found on the wiki, seem to be strictly vector-based. Is it even possible to mask an image, using another image, as a guide? If so, could somebody please post an example? I'm quite stumped.

Since my art is NSFW, I'll post a simplified diagram, showing what I'm going for:
Image

The basic idea is that there's a spotlight in the middle of the screen, and the character sprites are displayed as negative outlines outside of the spotlight, and fully colored art, inside the spotlight.
Lead programmer, and secondary artist at "Nosey Studios", I draw lots of butts! :D

User avatar
Ref
Party member
Posts: 690
Joined: Wed May 02, 2012 11:05 pm

Re: Stencil Help

Post by Ref » Sun Aug 18, 2019 1:42 am

Maybe something like this in reverse?
(Very old code so don't know how good it is)
Attachments
runningGirl.love
For what it'is worth
(146.34 KiB) Downloaded 84 times

User avatar
pgimeno
Party member
Posts: 2307
Joined: Sun Oct 18, 2015 2:58 pm

Re: Stencil Help

Post by pgimeno » Sun Aug 18, 2019 2:33 am

Welcome to the forums. By default, alpha is not taken into account when drawing to the stencil, only the geometry. That can be changed with a shader. For example, this shader makes every pixel with alpha < 0.5 to not be drawn to the stencil:

Code: Select all

vec4 effect(vec4 colour, Image texture, vec2 texpos, vec2 scrpos)
{
    vec4 pixel = Texel(texture, texpos) * colour;
    if (pixel.a < 0.5) discard;
    return pixel;
}
See attachment for an example using your graphics.
Attachments
spotlightExample.love
(2.7 KiB) Downloaded 87 times

User avatar
Alp
Prole
Posts: 7
Joined: Wed Jul 10, 2019 2:48 pm

Re: Stencil Help

Post by Alp » Sun Aug 18, 2019 1:18 pm

Ref wrote:
Sun Aug 18, 2019 1:42 am
Maybe something like this in reverse?
(Very old code so don't know how good it is)
Oh, that's a neat effect! It's not quite what I'm going for (a bit too noisy for my liking), but it looks great!
My art is strictly pixel-based, so I want really sharp lines.
pgimeno wrote:
Sun Aug 18, 2019 2:33 am
Welcome to the forums. By default, alpha is not taken into account when drawing to the stencil, only the geometry. That can be changed with a shader. For example, this shader makes every pixel with alpha < 0.5 to not be drawn to the stencil:

Code: Select all

vec4 effect(vec4 colour, Image texture, vec2 texpos, vec2 scrpos)
{
    vec4 pixel = Texel(texture, texpos) * colour;
    if (pixel.a < 0.5) discard;
    return pixel;
}
See attachment for an example using your graphics.
Ah! Thank you! That's EXACTLY what I was going for! So, you can use the alpha for stencils? Huh, good to know! I was under the assumption that stencils were strictly bitmask-based.

So, "pixel.a" is the value of the alpha channel for a given pixel, correct? Just making sure I understand what the code is doing.

For what it's worth, attempts were made, based on information from the wiki, but the closest I was able to get, was inverting the bit mask (black to white, and white to black), and I don't even understand how THAT worked! :P

I've attached my previous attempts, and the result, based on the code you provided. (They're NSFW, so fair warning.)
test_stencil2.love
NSFW
(15.05 KiB) Downloaded 89 times
test_stencil3.love
NSFW
(14.83 KiB) Downloaded 94 times
spotlight10.love
(64.28 KiB) Downloaded 93 times
"stencil2 + stencil3" are my attempts, but I couldn't quite get it right, after a few hours of struggling with it.
"spotlight10" is the result I was aiming for, based on your example. The curtains are a placeholder from Mario 2, for the moment.

Could the example code you provided be posted to the wiki? It would be VERY helpful for others, I imagine!

EDIT: The forums dropped my attachments, argh.
Lead programmer, and secondary artist at "Nosey Studios", I draw lots of butts! :D

User avatar
pgimeno
Party member
Posts: 2307
Joined: Sun Oct 18, 2015 2:58 pm

Re: Stencil Help

Post by pgimeno » Sun Aug 18, 2019 3:42 pm

Cool, and nice graphics!
Alp wrote:
Sun Aug 18, 2019 1:18 pm
Ah! Thank you! That's EXACTLY what I was going for! So, you can use the alpha for stencils? Huh, good to know! I was under the assumption that stencils were strictly bitmask-based.
A stencil is a buffer that holds numbers. It's not a bitmask, but you can apply a comparison method to it and the result is always a boolean which tells whether each corresponding pixel should be drawn or ignored. By default the stencil function sets it to 1, so I used "equal to 1" or "not equal to 1" as comparison functions.

Alp wrote:
Sun Aug 18, 2019 1:18 pm
So, "pixel.a" is the value of the alpha channel for a given pixel, correct? Just making sure I understand what the code is doing.
Yes, in GLSL you can address the elements of a vec4 with .xyzw or with .rgba (.x is the same as .r and so on).

If you don't know about shaders, there's a good introduction to the basics in the LÖVE blog: http://blogs.love2d.org/content/beginners-guide-shaders and the default shader used by LÖVE (very similar to the one I used) is here: https://love2d.org/wiki/love.graphics.newShader#Notes_2

Alp wrote:
Sun Aug 18, 2019 1:18 pm
Could the example code you provided be posted to the wiki? It would be VERY helpful for others, I imagine!
There's one example already which uses black for transparent instead of an alpha threshold: https://www.love2d.org/wiki/love.graphi ... encil_mask

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 23 guests