Unstable getImageData? (crash)

[miko]

Clearly a memory leak somewhere

I'm pretty sure there's no memory leak, it's just the way Lua handles user data.

Youare right in the sense you have described. It is a "leak" from the lua point of view.
The "workaround" is to put

Code: Select all

collectgarbage("step")

or

Code: Select all

collectgarbage("collect")

in love.update() - this fixes problem of memory usage (and crashing) for me.

[leiradel]

Clearly a memory leak somewhere

I'm pretty sure there's no memory leak, it's just the way Lua handles user data.

Youare right in the sense you have described. It is a "leak" from the lua point of view.
The "workaround" is to put

Code: Select all

collectgarbage("step")

or

Code: Select all

collectgarbage("collect")

in love.update() - this fixes problem of memory usage (and crashing) for me.

It's not really a leak from either C++ or Lua side, it's just regular memory management. See this thread on the Lua mailing list for a great discussion about the subject and the pros/cons of this workaround.

One of the proposed solution is to use custom allocators on the C/C++ side to make Lua aware of the total memory used:

Code: Select all

void* my_malloc(size_t sz, lua_State* L)
{
  void* p = lua_newuserdata(L, sz);
  lua_pushlightuserdata(L, p);
  lua_insert(L, -2);
  lua_settable(L, LUA_REGISTRYINDEX);
  return p;
}

void my_free(void* p, lua_State* L)
{
  lua_pushlightuserdata(L, p);
  lua_pushnil(L);
  lua_settable(L, LUA_REGISTRYINDEX);
}

and change code to always allocate and free memory using those functions. C++ instances can use the placement new on the pointer returned by my_malloc. The drawbacks of this workaround are:

• It's slower than malloc/free because you're asking Lua to allocate/free memory (which in turn calls malloc/free) and because of the extra handling to put/remove the user data into/from the Lua registry. But since malloc is already slow, it might not be an issue.
• There's no realloc, if you need to increase/decrease the size of the memory a pointer points to you'll have to allocate memory, copy the contents of the old pointer to the new pointer and free the old pointer. Of course it's just what realloc does when the current block cannot be made larger.
• It does not solve the problem if the allocation is made by an external library, unless you patch it to use my_alloc/my_free

Lua 5.2 added an emergency garbage collector. The documentation was not updated, but I think that what it does is to make it safe to execute the garbage collector inside your lua_Alloc function if the allocation failed so that the GC frees some memory and you can retry your allocation. I don't think this will solve this particular problem since page swaps will begin much before an allocation fails, probably making things so slow you'll have to kill LÖVE. But taking note of actual memory usage in your lua_Alloc and limiting it in some reasonable way may solve it.

Cheers,

Andre

[Twinmold]

Let me try to guess what the problem is...

When you make a call to fb:getImageData() a new image data is created, which has all pixels from the frame buffer so the memory requirements for it is width*height*bpp (where bpp is probably 4.) But on the Lua side of things what is created is a Proxy userdata, which takes only a handful of bytes and points to the actual image data object.

So the problem is, Lua doesn't see a need to trigger the garbage collection because for it each image data object takes only the size of a Proxy when in fact the each of these proxies point to an image data which takes far more memory. So the RAM is exhausted and eventually the computer begins to swap pages, slows down, and crashes (Windows, Linux will probably do something wiser.)

The need to inform Lua about the total size of an userdata, which in this case would include the size of the image data, comes up in its mailing list from time to time, but it was never added to the API. I don't remember if there's an workaround for that besides always allocating memory through lua_newuserdata and using luaL_ref/unref on them to avoid them being collected when still being pointed to by other objects.

In this specific case, maybe fb:getImageData() could optionally accept an image data parameter the same size of the frame buffer, which would be used to get the frame buffer pixels instead of creating a new image data?

I'm surprised that I actually get it. Nice explanation, thanks.

Youare right in the sense you have described. It is a "leak" from the lua point of view.
The "workaround" is to put

Code: Select all

collectgarbage("step")

or

Code: Select all

collectgarbage("collect")

in love.update() - this fixes problem of memory usage (and crashing) for me.

[/size]
That surely does the trick. After a couple of tests -just checking the memory usage with collectgarbage(count)-, it seems there's actually no memory tradeoff in using collectgarbage() every step, even with a full collect. So, any caveats about just doing so?

[miko]

That surely does the trick. After a couple of tests -just checking the memory usage with collectgarbage(count)-, it seems there's actually no memory tradeoff in using collectgarbage() every step, even with a full collect. So, any caveats about just doing so?

In theory garbage collector should collect unused objects only when necessary (it is smart enough if it knows how much memory every object uses). If you force full/partial collection every love.update(), you are using more CPU. On the other hand, if you do not create too many objects in every step, collectgarbage() should not take that long to run.
In this case, running collectgarbage() is the only solution to prevent excessive memory usage and crashing of the application.

[Ghuntar]

Haha, tested it on a "real" laptop (quad core CPU, 8Gb RAM, etc...), there is no slow down, but the memory take up to 3.8Gb (almost 4Gb I would say) then stop to grow. More, only one CPU is used but at 95%.
Are these limits given from löve or lua ?