How to save and reload a very large amount of data?

Questions about the LÖVE API, installing LÖVE and other support related questions go here.
Forum rules
Before you make a thread asking for help, read this.
amorphia
Prole
Posts: 8
Joined: Sat May 20, 2017 5:21 pm

How to save and reload a very large amount of data?

Post by amorphia » Sat May 20, 2017 5:35 pm

Hi all!

I need to be able to save and load a very large amount of data, because I want to be able to do in-game replays of games that have already been played.

So as I draw each object, I save the time, location, identity, etc, to a table, and when the game is over, I use bitser (https://github.com/gvx/bitser) to save the table. That works fine, even though the files are obviously large.

Then, in the next game, when I need to replay the recording, I load the saved file. The problem is, if the file is large (1Mb is OK, 3Mb is not) then the bitser.loadLoveFile() just crashes. Not a lua error message, a full on Windows "this program has stopped working" crash.

Is bitser not designed for large files? Is there a better way to do this? Any tips much appreciated.

Cheers,

Amorphia

User avatar
raidho36
Party member
Posts: 1685
Joined: Mon Jun 17, 2013 12:00 pm

Re: How to save and reload a very large amount of data?

Post by raidho36 » Sat May 20, 2017 6:03 pm

It's possible that bitser uses way to much memory internally, overloading Lua's capability. In the end, your computer memory can only store so much data at a time. The solution is to use some sort of streaming. Write data to disk frequently but in small packets and immediately reuse the memory to store next packet. During replay, only load few packets at a time, reuse memory from packets you already played back.

User avatar
zorg
Party member
Posts: 2379
Joined: Thu Dec 13, 2012 2:55 pm
Location: Absurdistan, Hungary
Contact:

Re: How to save and reload a very large amount of data?

Post by zorg » Sat May 20, 2017 6:18 pm

I'd wager that Robin knows better than to inflate 3 megabytes of data into over 2 gigabytes in-memory as to crash luaJIT...
3MB is not that big of an amount to be honest, maybe it's a bug in bitser, or your usage of it, can't tell without seeing code.
Me and my stuff :3True Neutral Aspirant. Why, yes, i do indeed enjoy sarcastically correcting others when they make the most blatant of spelling mistakes. No bullying or trolling the innocent tho.

amorphia
Prole
Posts: 8
Joined: Sat May 20, 2017 5:21 pm

Re: How to save and reload a very large amount of data?

Post by amorphia » Sat May 20, 2017 9:11 pm

Thanks a lot for the suggestions folks. If I'm going to have to write code for streaming, I might as well roll my own routines for all of this - I'd prefer to do neither. Here is minimal code which recreates the bug. Run it, wait a few seconds, press escape, run it again, press r. Advice to whether this is my bug or bitser's much appreciated!

Code: Select all

bitser = require 'bitser'

drawReg = {}
regIndex = 1

function love.load()
	love.filesystem.setIdentity('megaAttackGame');
end

function love.update(dt)
	if love.keyboard.isDown("escape") then
		success = bitser.dumpLoveFile('replay.dat', drawReg )
		love.event.quit()
	elseif love.keyboard.isDown("r") then
		replayData = bitser.loadLoveFile('replaySave2.dat')
	end
end

function love.draw(dt)
	for i=1,500 do
		registerVisible( 1, 1, "thing")
	end
end

function registerVisible( xPos, yPos, thingToDraw )
	drawReg[regIndex] = {
		timeStamp = os.clock(),
		x = xPos,
		y = yPos,
		t = thingToDraw
	}
	regIndex = regIndex + 1
	--Full version of function draws the thing as well
end

User avatar
zorg
Party member
Posts: 2379
Joined: Thu Dec 13, 2012 2:55 pm
Location: Absurdistan, Hungary
Contact:

Re: How to save and reload a very large amount of data?

Post by zorg » Sat May 20, 2017 10:39 pm

Code: Select all

success = bitser.dumpLoveFile('replay.dat', drawReg )
That function does not return any values, so success will always be nil.

Also, the two filename strings should be the same to get the error.

As for the actual error's reason... i'm not sure.
it works for 500 and 5000 iterations.
for 50 000 it goes over the deserialize_value function 550003 times, and returns the correct amount of items.
for 500 000, it flat out dies in the deserialize_value function.

My best bet, open an issue on github, that'd be the fastest way to get help regarding this.

Also, you don't want to detect keypresses for this kind of thing in update... it might run the function more than once (it happened with me since i held down the keys a bit long); you want to use the love.keypressed callback instead.
Me and my stuff :3True Neutral Aspirant. Why, yes, i do indeed enjoy sarcastically correcting others when they make the most blatant of spelling mistakes. No bullying or trolling the innocent tho.

User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

Re: How to save and reload a very large amount of data?

Post by Robin » Mon May 29, 2017 2:05 pm

I'm looking into this, but I'm quite busy at the moment, so it might take a few days or longer before I get around to fixing this issue. There might be some inherent limitation we're running into, in which case I'll update the documentation.
Help us help you: attach a .love.

amorphia
Prole
Posts: 8
Joined: Sat May 20, 2017 5:21 pm

Re: How to save and reload a very large amount of data?

Post by amorphia » Mon May 29, 2017 4:02 pm

Thanks Robin, and everyone else who's been helping!

If it does turn out this isn't going to be a job for Bitser, I have made a plan B. I would output a file actually in .lua format, declaring a table, and then just require the file to re-import it. The file would be bigger than necessary as it would be text, but that's not a huge problem for my application. Is that a silly way to do it?

User avatar
zorg
Party member
Posts: 2379
Joined: Thu Dec 13, 2012 2:55 pm
Location: Absurdistan, Hungary
Contact:

Re: How to save and reload a very large amount of data?

Post by zorg » Mon May 29, 2017 6:46 pm

amorphia wrote:
Mon May 29, 2017 4:02 pm
Thanks Robin, and everyone else who's been helping!

If it does turn out this isn't going to be a job for Bitser, I have made a plan B. I would output a file actually in .lua format, declaring a table, and then just require the file to re-import it. The file would be bigger than necessary as it would be text, but that's not a huge problem for my application. Is that a silly way to do it?
You can absolutely do that, something like (semi-pseudocode)

Code: Select all

function save(path, events)
local s = { "return {" }
for i,v in ipairs(events) do
-- You'd handle different types of vars differently, like saving "true" or "false" strings for bools, enclosing strings with ', etc...
table.insert(s,('%16.16g'):format(v)) -- for numbers, for example.
end
-- close off the table.
table.insert("}")
-- then write it out into a file, using table.concat or not; with will waste a bit of memory, without will maybe take a bit longer (writing things at once vs. everything sequentially, depends on buffers and stuff, not that important)
end
Me and my stuff :3True Neutral Aspirant. Why, yes, i do indeed enjoy sarcastically correcting others when they make the most blatant of spelling mistakes. No bullying or trolling the innocent tho.

alloyed
Citizen
Posts: 80
Joined: Thu May 28, 2015 8:45 pm
Contact:

Re: How to save and reload a very large amount of data?

Post by alloyed » Tue May 30, 2017 12:24 am

I've also experienced crashes in bitser with large tables, but I couldn't reproduce it outside of my game so I just designed around it. in my case it didn't seem to be related to serialized file size: the test files I dumped before each crash were all over the place when it came to size, and maybe more importantly wouldn't crash if I made a program that just loaded the dumped files, and then serialized them back.

When it comes to streaming using bitser, the lazy approach I took was just to adopt a metaformat: when I read and write, I read and write chunks that represent a full bitser string. The first 64 bits are the length of the chunk, and then we read the full chunk using that and pass the resulting string to bitser. This works fine, although I haven't tested it against file corruption etc. I guess if it became a problem we can checksum each chunk.

amorphia
Prole
Posts: 8
Joined: Sat May 20, 2017 5:21 pm

Re: How to save and reload a very large amount of data?

Post by amorphia » Wed May 31, 2017 8:45 am

Great, thanks everyone!

Post Reply

Who is online

Users browsing this forum: MrFariator, Stifu and 11 guests