Questions on Saving data aka serializing table [SOLVED]

General discussion about LÖVE, Lua, game development, puns, and unicorns.
Post Reply
Muris
Party member
Posts: 131
Joined: Fri May 23, 2014 9:18 am

Questions on Saving data aka serializing table [SOLVED]

Post by Muris » Tue Jan 13, 2015 2:15 pm

Hello and sorry for asking something trivial but I am still quite new to LUA

I tried couple of the lua serializing tools to serialize my object: https://github.com/gvx/Smallfolk and smallfolk failed to deserialize on "exp" = 0, it didn't like 0 as a number.

Second one I tried was https://love2d.org/wiki/Tserialwhich for some reason couldn't deserialize the stuff it serialized.

The third one I tried actually works just fine, but it returns unsafe lua-file: https://github.com/gvx/Ser
Basically the output is something as following in ser, where it just has a huge list of characters and their stats:

Code: Select all

_[36] = {"0", "300000"}

_[17] = {classType = "Ranger", important = true, expGain = 50, alias = "Peter", npc = false, spi = 77, vit = 155, lvl = 101, str = 112, skills = _[36], img = "8", int = 99, exp = 0, luc = 146, speed = 5, name = "Peter", dex = 230}

_[1] = {["4"] = _[2], test2 = _[3], ["5"] = _[4], test7 = _[5], test4 = _[6], test9 = _[7], test6 = _[8], ["7"] = _[9], test8 = _[10], ["1"] = _[11], ["6"] = _[12], test1 = _[13], test5 = _[14], ["3"] = _[15], test3 = _[16], ["2"] = _[17], test10 = _[18], test12 = _[19], test11 = _[20]}

return {units = _[1]}
So the units just have an id, and which refers to the unit, which has a list of skills on their disposal. I used strings for identifiers, since I was first using JSON to describe my character data.

So the way I am actually loading is by using loadstring, which could do a lot of unwanted stuff. So I was thinking if I run something like:

Code: Select all

local brackets = serString:find("[()]") 	
if not brackets then
  local f = assert(loadstring( serString ))
  local saveState = f()
  ... do my stuff
end
Now this leaves me to the question: If I take of brackets (), and make sure that I do not use brackets inside any of my variables, would it lead to complete safe code? Like can anyone do anything that can break stuff with the reserved words such as return, end etc? The reason why I am trying to remove () is to detect function declarations / function calls, although someone can still overwrite global function names I think like love.draw = nil

I am able to accept that the game crashes if the savefile is incorrect, but not stuff like delete files from harddrive by modifying the save file etc. Also since I am kinda trying to get this working in android at the same time, I think that I should be a bit more caution about these things.
Last edited by Muris on Tue Jan 13, 2015 4:25 pm, edited 1 time in total.

User avatar
kikito
Inner party member
Posts: 3153
Joined: Sat Oct 03, 2009 5:22 pm
Location: Madrid, Spain
Contact:

Re: Questions on Saving data aka serializing table

Post by kikito » Tue Jan 13, 2015 2:43 pm

You can always put your loadstring / ser code in a sandbox with sandbox.lua:

Code: Select all

local sandbox = require 'sandbox'
...
local saveState = assert(sandbox.run(serString))
...
sandbox.lua will prevent nasty things from being executable inside serString. The most harmful thing they will be able to do is block your game (with an infinite loop - we can prevent that in Lua 5.1, but not in LuaJIT).
When I write def I mean function.

Muris
Party member
Posts: 131
Joined: Fri May 23, 2014 9:18 am

Re: Questions on Saving data aka serializing table

Post by Muris » Tue Jan 13, 2015 4:31 pm

kikito wrote:You can always put your loadstring / ser code in a sandbox with sandbox.lua:

Code: Select all

local sandbox = require 'sandbox'
...
local saveState = assert(sandbox.run(serString))
...
sandbox.lua will prevent nasty things from being executable inside serString. The most harmful thing they will be able to do is block your game (with an infinite loop - we can prevent that in Lua 5.1, but not in LuaJIT).
Thanks, this definitely does seem to cover the case of changing global values.

I am still leaving the checking for brackets '(' and ')' for just in case, although it is probably unneccessary. I am just thinking about some odd case where someone defines a function in one of the table values which then changes the global values somehow when being called in some random case, although it shouldn't happen since everything should be either tables / strings or numbers, and thus it should always throw an exception when handling a function as other type of field. Maybe adding some validation for the data would cover most of this.

User avatar
kikito
Inner party member
Posts: 3153
Joined: Sat Oct 03, 2009 5:22 pm
Location: Madrid, Spain
Contact:

Re: Questions on Saving data aka serializing table [SOLVED]

Post by kikito » Tue Jan 13, 2015 5:47 pm

Even if they define a function inside loadstring, it won't be able to modify any global variable. This is due to the way the sandbox is built - functions declared in the loadstring will be inside a "protected scope", not in the global one.
When I write def I mean function.

Muris
Party member
Posts: 131
Joined: Fri May 23, 2014 9:18 am

Re: Questions on Saving data aka serializing table [SOLVED]

Post by Muris » Tue Jan 13, 2015 6:25 pm

kikito wrote:Even if they define a function inside loadstring, it won't be able to modify any global variable. This is due to the way the sandbox is built - functions declared in the loadstring will be inside a "protected scope", not in the global one.
This is indeed quite nice, just tested it with following line of code:

Code: Select all

sandbox.run( "return function() print(\"test\") end")()
And it really is like you said, the whole global space is unreachable from sandboxed code, it cannot find print.

User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

Re: Questions on Saving data aka serializing table [SOLVED]

Post by Robin » Tue Jan 13, 2015 8:50 pm

In the case you cannot trust the source of the data, there is actually another library made especially for this purpose: Smallfolk. It's output also tends to be smaller than Ser's, and you use smallfolk.loads(string) instead of loadstring(string)().
Help us help you: attach a .love.

Muris
Party member
Posts: 131
Joined: Fri May 23, 2014 9:18 am

Re: Questions on Saving data aka serializing table [SOLVED]

Post by Muris » Tue Jan 13, 2015 8:57 pm

Robin wrote:In the case you cannot trust the source of the data, there is actually another library made especially for this purpose: Smallfolk. It's output also tends to be smaller than Ser's, and you use smallfolk.loads(string) instead of loadstring(string)().
smallfolk wasnt able to deserialize a case with having a number 0 as a value. I had an object with exp = 0, and the deserializer threw exception on deserializing the string that it serialized.

Here is a code that it fails to work with:

Code: Select all

local testcase = smallfolk.dumps({ a = 0 })
print( smallfolk.loads(testcase).a )
Also 0. something doesnt work, it fails with:

Code: Select all

local testcase = smallfolk.dumps({ a = 0.12 })
print( smallfolk.loads(testcase).a )
And same with negative -0.somehting:

Code: Select all

local testcase = smallfolk.dumps({ a = -0.15 })
print( smallfolk.loads(testcase).a )


Also for some reason the object that I tried to serialize with TSerialize also failed to deserialize it its own serialization

edit added a testcase that the serializer cannot complete.
edit2: added another case that the deserializer doesnt work with
Last edited by Muris on Tue Jan 13, 2015 9:28 pm, edited 1 time in total.

User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

Re: Questions on Saving data aka serializing table [SOLVED]

Post by Robin » Tue Jan 13, 2015 9:26 pm

That is a serious bug that I'll fix right away. Can't believe I didn't test for 0!

EDIT: fix on github!
Help us help you: attach a .love.

Muris
Party member
Posts: 131
Joined: Fri May 23, 2014 9:18 am

Re: Questions on Saving data aka serializing table [SOLVED]

Post by Muris » Tue Jan 13, 2015 9:43 pm

Robin wrote:That is a serious bug that I'll fix right away. Can't believe I didn't test for 0!

EDIT: fix on github!
Nice thank you for the fast fix, which seems to also work with the 0. something aswell. Now it seems to be able to deserialize the character stats. Also it seems to be able to distinguish 0 and "0", so I think I won't have any other problems on using it.

User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

Re: Questions on Saving data aka serializing table [SOLVED]

Post by Robin » Tue Jan 13, 2015 10:29 pm

Good to hear! If you run into any further problems, using either Smallfolk or Ser, I would appreciate you contact me, either directly, or by making an issue on the Github repository (I think you need a Github account for that), that way I can fix the problem as soon as possible. :)
Help us help you: attach a .love.

Post Reply

Who is online

Users browsing this forum: bobbymcbobface, Google [Bot] and 6 guests