Page 1 of 2
bitser: faster, harder, better serialization with LuaJIT
Posted: Tue Feb 16, 2016 11:56 am
by Robin
Oops I did it again
I played with serialization, got lost in the game
Oh baby, baby
bitser is the ultimate serialization library for LÖVE:
- Uses LuaJIT to be superfast and superefficient
- Just like Lady and binser, it supports registering resources (like Image, Source, anything) and classes (MiddleClass, SECL, hump.class, Slither)
- It's secure: you can use it to serialize over teh interwebs
- LÖVE bonus: you can load serialized values from FileData objects as well as from strings, which means loading gets even faster
- Seriously, try it out, if you run the repo with LÖVE you get a benchmark with nice charts that you can compare if you download other serialization libraries to it.
Re: bitser: faster, harder, better serialization with LuaJIT
Posted: Tue Feb 16, 2016 12:21 pm
by qaisjp
Nice work! What kind of format does it serialize to?
Re: bitser: faster, harder, better serialization with LuaJIT
Posted: Tue Feb 16, 2016 12:49 pm
by Robin
It uses a custom binary format that is optimized for size and writing time.
Here's an overview of each initial byte:
Code: Select all
0 * * * * * * * -> small int
1 0 * * * * * * -> small ref
1 1 0 * * * * * -> small string [data:byte * rest]
1 1 1 0 * * * * -> small res [label:byte * rest]
1 1 1 1 0 0 0 0 -> table [len:number] [value:any] * len [keylen:number] ([key:any] [value:any]) * keylen
1 1 1 1 0 0 0 1 -> resource [label:string]
1 1 1 1 0 0 1 0 -> instance [class:string] [data:table]
1 1 1 1 0 0 1 1 -> reference [idx:number]
1 1 1 1 0 1 0 0 -> string [len:number] [data:byte * len]
1 1 1 1 0 1 0 1 -> long [data:byte * 4]
1 1 1 1 0 1 1 0 -> float [data:byte * 8]
1 1 1 1 0 1 1 1 -> nil
1 1 1 1 1 0 0 0 -> false
1 1 1 1 1 0 0 1 -> true
1 1 1 1 1 0 1 0 -> short [data:byte * 2]
1 1 1 1 1 0 1 1 -> N/A
1 1 1 1 1 1 0 0 -> N/A
1 1 1 1 1 1 0 1 -> N/A
1 1 1 1 1 1 1 0 -> N/A
1 1 1 1 1 1 1 1 -> N/A
Type bytes 251-255 are reserved for future extension.
Re: bitser: faster, harder, better serialization with LuaJIT
Posted: Wed Feb 17, 2016 12:47 am
by prixt
Is it possible to register custom classes that don't use libraries you mentioned? I prefer to create classes on a case by case basis, so don't use outside libraries.
Re: bitser: faster, harder, better serialization with LuaJIT
Posted: Wed Feb 17, 2016 1:27 am
by pgimeno
Robin wrote:- It's secure: you can use it to serialize over teh interwebs
Yay! Security is the very reason I've used a JSON library instead of any of the other serializers out there, in my only project so far (which didn't use self-referential tables or any other stuff unsupported by JSON). Each and every serializer I've found that was able to save native Lua data in textual form, loaded the data through loadstring.
The fact that it outputs to binary only is a problem for me to adopt it, though. I'm not a big fan of obscurity in savegames in most situations. One should not need to use a savegame editor just to make a few changes.
However, I like the idea for complex games where there's a lot of data and storing it in a compact way is best.
Re: bitser: faster, harder, better serialization with LuaJIT
Posted: Wed Feb 17, 2016 1:42 am
by monolifed
does small int work like variable int in protobuf (uses less bytes for small integers)
Re: bitser: faster, harder, better serialization with LuaJIT
Posted: Wed Feb 17, 2016 5:04 am
by airstruck
pgimeno wrote:Each and every serializer I've found that was able to save native Lua data in textual form, loaded the data through loadstring.
You could use setfenv with loadstring, it should be reasonably secure.
Code: Select all
local f = loadstring(serialized)
setfenv(f, {})
local deserialized = f()
I don't think it could do any real damage sandboxed like that, although it could try to be annoying by hanging the game or something (someone please correct me if I'm wrong).
Re: bitser: faster, harder, better serialization with LuaJIT
Posted: Wed Feb 17, 2016 10:16 am
by Robin
prixt wrote:Is it possible to register custom classes that don't use libraries you mentioned? I prefer to create classes on a case by case basis, so don't use outside libraries.
It depends on how exactly you use it. Bitser has to heuristically detect how the class works and how to construct instances when loading. If the class is the metatable of instances, for example, it would work no problem, because bitser would think you're using hump.class. When registering a class you can also give a classkey and deserializer:
Code: Select all
bitser.registerClass('YourClass', YourClass, 'classkey', function(instance, class) return --[[make instance an instance of class]] end)
pgimeno wrote:The fact that it outputs to binary only is a problem for me to adopt it, though. I'm not a big fan of obscurity in savegames in most situations. One should not need to use a savegame editor just to make a few changes.
That's a good point. That's maybe a good idea for another library, that maximizes readability at the expense of size, speed and security. Originally I wrote Lady for that, but Lady is not very readable, making hand-editing it less fun.
pgimeno wrote:However, I like the idea for complex games where there's a lot of data and storing it in a compact way is best.
Yeah, that's more the intended use case.
ingsoc451 wrote:does small int work like variable int in protobuf (uses less bytes for small integers)
Yes. Integers from -27 to 100 are stored in the type byte.
Re: bitser: faster, harder, better serialization with LuaJIT
Posted: Fri Feb 19, 2016 1:55 am
by bakpakin
Wow. That is quality.
For luajit needs, binser might be obsolete. All that work I put into making sure numbers serialized correctly, and you go and just use ffi to copy memory.
The links for other libraries for the benchmark are broken, though. Replacing "github.com" in the urls with "raw.githubusercontent.com" should do the trick, I think.
Great work. This looks like it has applications outside of LOVE. I might use this with
luvit.
Re: bitser: faster, harder, better serialization with LuaJIT
Posted: Sat Feb 20, 2016 2:39 pm
by Robin
Thanks!
Huh, the links worked for me. Maybe I should replace them then.