Coverage reports in LOVE

General discussion about LÖVE, Lua, game development, puns, and unicorns.
Post Reply
User avatar
CaptainMaelstrom
Party member
Posts: 157
Joined: Sat Jan 05, 2013 10:38 pm

Coverage reports in LOVE

Post by CaptainMaelstrom » Fri Nov 15, 2019 3:47 pm

Has anybody successfully created coverage reports working in LOVE?

I've tried luacov (https://keplerproject.github.io/luacov/) but been unsuccessful. I can set the require paths so that the luacov lib loads ok, I can start it and know it's collecting stats, but I haven't been able to figure out how to get it to print a coverage report file. Even with the supplied tick command. When I try to love.filesystem.write(luacov.stats, 'cov.txt'), I end up getting errors from other parts of the library (hook.lua).

I'll keep trying, maybe end up rolling my own coverage reports like I did for tests. But if anybody has any insight on generating coverage reports for LOVE code I'd be happy to hear them.

User avatar
ivan
Party member
Posts: 1564
Joined: Fri Mar 07, 2008 1:39 pm
Contact:

Re: Coverage reports in LOVE

Post by ivan » Fri Nov 15, 2019 4:52 pm

Love2D uses LuaJIT so you have to make sure that tool is compatible with LuaJIT.
I don't see LuaJIT mentioned anywhere in the docs.
Generally speaking, profiling with LuaJIT is hard because it's super optimized and some hooks are not triggered at all.
I would love to see a better way to profile Love2D, until then you have to use an inferior tool like my profiler.
Hope that helps.

Fuzzlix
Citizen
Posts: 60
Joined: Thu Oct 13, 2016 5:36 pm

Re: Coverage reports in LOVE

Post by Fuzzlix » Fri Nov 15, 2019 6:20 pm

CaptainMaelstrom wrote:
Fri Nov 15, 2019 3:47 pm
.. but I haven't been able to figure out how to get it to print a coverage report file.
If i remember correctly, it writes/append to a somehow binary datafile and you have to run luacov from commandline afterwards to generate a text file.

User avatar
raidho36
Party member
Posts: 2063
Joined: Mon Jun 17, 2013 12:00 pm

Re: Coverage reports in LOVE

Post by raidho36 » Fri Nov 15, 2019 6:38 pm

Generally, for debugging you want to use interpreted Lua. Just swap out the DLL.

JoshGrams
Prole
Posts: 33
Joined: Sat May 27, 2017 10:58 am

Re: Coverage reports in LOVE

Post by JoshGrams » Sat Nov 16, 2019 12:39 am

LuaCov works fine for me under Windows in 11.2. I'm doing:

Code: Select all

require('luacov.runner').init({
	runreport = true,
	deletestats = true,
	exclude = {
		"^tests/.*"
		-- etc...
	}
})
Then after running my tests I just call love.event.quit as usual, and it writes the coverage report to luacov.report.out (I don't have to do anything special to make that happen). I haven't messed with this setup in ages so I don't remember if I had to do anything special or hack luacov at all to get it running (I don't think so).

User avatar
CaptainMaelstrom
Party member
Posts: 157
Joined: Sat Jan 05, 2013 10:38 pm

Re: Coverage reports in LOVE

Post by CaptainMaelstrom » Sun Nov 17, 2019 5:12 am

ivan and raidho36, I hadn't considered the added difficulty of testing/covering LuaJIT so thanks for making me aware of that extra gotcha.

JoshGrams, thanks for showing me that. I found more init options here:
https://keplerproject.github.io/luacov/ ... ml#modules

I'm now setting my own statsfile and reportfile paths so I can find them both. My stats file has lines like this:

Code: Select all

57:extends/debug.lua
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 336 0 336 336 336 0 336 336 0 0 0 0 0 0 336 
But my report file is:

Code: Select all

==============================================================================
Summary
==============================================================================

File  Hits Missed Coverage
--------------------------
Total 0    0      0.00%
I haven't excluded anything so I expected all files to get reported on. So I wonder why it's not showing me a report like:

Code: Select all

==============================================================================
test.lua
==============================================================================
 1 if 10 > 100 then
*0    print("I don't think this line will execute.")
   else
 1    print("Hello, LuaCov!")
   end
I must be missing some other init option. Anybody know how to interpret the stats file? Do you think swapping the .dlls like raidho36 suggested will help my situation? I'd rather test LuaJIT since that's what I'm deploying.

User avatar
raidho36
Party member
Posts: 2063
Joined: Mon Jun 17, 2013 12:00 pm

Re: Coverage reports in LOVE

Post by raidho36 » Sun Nov 17, 2019 8:56 am

LuaJIT and vanilla Lua are functionally identical. That is, if you don't use LuaJIT-specific functionality (such as C data) you should be fine with just swapping the DLL. Beyond running slower, there will be no difference in the way the code runs.

Well actually there will be subtle differences. For instance, LuaJIT will use CPU instructions for math operations where applicable, and those don't follow IEEE as a rule. Particular pitfall I've experienced is that using min or max with NaN as one of operands will not ignore the NaN and may return it if it was the right operand of the binary CPU instruction, and order of these operands is pretty random in compiled code.

User avatar
pgimeno
Party member
Posts: 2211
Joined: Sun Oct 18, 2015 2:58 pm

Re: Coverage reports in LOVE

Post by pgimeno » Sun Nov 17, 2019 1:06 pm

Maybe executing it with jit.off() helps?

User avatar
ivan
Party member
Posts: 1564
Joined: Fri Mar 07, 2008 1:39 pm
Contact:

Re: Coverage reports in LOVE

Post by ivan » Sun Nov 17, 2019 1:17 pm

I'd rather test LuaJIT since that's what I'm deploying
When profiling, you will ALWAYS be profiling the Lua code, not the underlying binaries.

User avatar
CaptainMaelstrom
Party member
Posts: 157
Joined: Sat Jan 05, 2013 10:38 pm

Re: Coverage reports in LOVE

Post by CaptainMaelstrom » Sun Nov 17, 2019 7:16 pm

I went to my love2d directory and swapped the existing LOVE 11.3 file called "lua51.dll" with the "lua5.1.dll" downloaded from:
https://sourceforge.net/projects/luabin ... p/download

Note: I'm using the 64-bit zipped Windows version of LOVE 11.3.

The report file looks the same -- empty, but now my print statements are not printing to console. Did I swap the right dlls?

I also tried jit.off() and my report file is still empty. I tried this after reverting the .dlls.

I really don't want to give up on coverage reports, so if anyone else has any ideas of things to try or alternatives to luacov even, I'm all ears.

Thanks all.

Post Reply

Who is online

Users browsing this forum: No registered users and 24 guests