A hacky replacement for stencils? (Lovebrew/Löve potion)

[Bananicorn]

Hey there fellow lövers!
So, the inofficial (but nonetheless genius) 3ds port of Löve doesn't support stencils,
but I love the idea of making a game for 3ds, even if it's only homebrew.

Is there any way to emulate stencils without support for, well stencils and shaders?

I thought about using love.graphics.setBlendMode on an intermediate canvas, or two and somehow hacking my way around it, but it seems I misread the blendMode formulas found here: https://love2d.org/wiki/BlendMode_Formulas

I'll also attach the .love file to give you an idea of what I want to achieve: objects being occluded in an isometric (dimetric) environment.
I've tried just re-drawing the tiles over the object in question, but that needs to also redraw the tiles in front of it, could overdraw other objects, etc.

The Löve file is also full of other errors, bugs and smaller and bigger problems, but please ignore that.

The .love example here runs correctly, but I'd need to replace this simple shader here with something different that is supported by LövePotion:

Code: Select all

	local mask_shader = LG.newShader([[
		extern Image mask;
		extern float mask_size;
		extern vec2 canvas_origin;

		vec4 effect(vec4 color, Image texture, vec2 texture_coords, vec2 screen_coords){
			vec2 mask_coords = screen_coords - canvas_origin;
			vec2 mask_uv = mask_coords / mask_size;
			vec4 mask_pixel = Texel(mask, mask_uv);
			return vec4(color.rgb, color.a - mask_pixel.a);
		}
	]])
	mask_shader:send( 'mask', c )
	mask_shader:send( 'mask_size', cw )
	mask_shader:send( 'canvas_origin', {x - cw / 2, y - cw / 2} )

I'm at my wits end right now, maybe someone can tell me what I need is impossible, or point me to a löve feature I've not seen yet (but which is supported by LöveBrew)

[pgimeno]

It can be done but it's tricky.

You need to do the masking in a canvas, and the mask is applied with multiplicative mode. You need to switch blend modes several times.

- Initialize the canvas to transparent black, i.e. love.graphics.clear(0,0,0,0)
- To draw the shape that you're going to cut, use normal blend mode (alpha, alphamultiply).
- The mask must be white but with premultiplied alpha. That's one tricky part. Basically, if your mask is made of only fully opaque or fully transparent pixels (which is what a stencil can do, anyway), it means that all pixels that are fully transparent also need to be black. If you're unsure you can always use another canvas to ensure it's premultiplied, but that's probably overkill.
- Draw the mask to the canvas using multiplicative blend mode (multiply, premultiplied).
- Finally, draw the canvas to the screen using premultiplied alpha mode (alpha, premultiplied).
- Don't forget to restore normal blend mode when you're done (alpha, alphamultiply).

Code: Select all

-- Create a 7x1 image for a hue gradient as an example of an image to cut
local bg = love.graphics.newImage(love.image.newImageData(7, 1, "rgba8",
     "\255\000\000\255" -- Red
  .. "\255\255\000\255" -- Yellow
  .. "\000\255\000\255" -- Green
  .. "\000\255\255\255" -- Cyan
  .. "\000\000\255\255" -- Blue
  .. "\255\000\255\255" -- Magenta
  .. "\255\000\000\255" -- Red
))
-- Load a mask from disk
local mask = love.graphics.newImage('publicdomainq-perplexed-reduced-mask.png')

local canvas = love.graphics.newCanvas()

love.graphics.setBackgroundColor(0.5, 0.5, 0.5)

function love.draw()
  -- Draw to the canvas to do the masking
  love.graphics.setCanvas(canvas)

  -- The canvas needs to be initialized to transparent black for this to work
  love.graphics.clear(0, 0, 0, 0)

  -- Draw a hue gradient as background, stretched to 180x180
  love.graphics.draw(bg, 400, 450, 0, 180/7, 180/1, 7/2, 1/2)

  -- Do the masking here
  love.graphics.setBlendMode("multiply", "premultiplied")
  love.graphics.draw(mask, 400, 450, 0, 1, 1, mask:getWidth()/2, mask:getHeight()/2)
  -- Disable drawing to the canvas
  love.graphics.setCanvas()

  -- Draw the canvas to the screen
  -- With rare exceptions, canvases need to be drawn in premultiplied mode
  love.graphics.setBlendMode("alpha", "premultiplied")
  love.graphics.draw(canvas)

  -- Draw the mask in replace mode so that you can see the pixels (because the
  -- alpha will be ignored by the screen)
  love.graphics.setBlendMode("replace", "premultiplied")
  love.graphics.draw(mask, 500, 100)

  -- Restore normal blend mode
  love.graphics.setBlendMode("alpha", "alphamultiply")
  -- Draw the background
  love.graphics.draw(bg, 150, 100, 0, 180/7, 180/1)
end

Edit: Forgot to say that the mask must be white where alpha is maximum. Edited the description accordingly.

[Bananicorn]

That would've taken me an eternity to figure out, thank you!
I've yet to adapt it to my code and try it on the actual hardward, but I'll get back to you once I do!
That would make this the second game of mine in whose credits you'd appear (provided you don't mind).
Well, if I finish it, that is^^

[pgimeno]

Of course I don't mind Would you mind to use my actual name, Pedro Gimeno?

One clarification: white with premultiplied alpha means that each colour component must equal the alpha component. So basically the R layer, the G layer and the B layer are all copies of the A layer. That also lets you use partially transparent masks (which stencils don't).