Ser: a new old serialization library

[adnzzzzZ]

Oh snap, I always forget to update libraries. Thanks for pointing that out.

[ivan]

Nice job, Robin I like the lib a lot.
Hope you don't mind but I stole your line:
"local oddvals = {[tostring(1/0)] = '1/0', [tostring(-1/0)] = '-1/0', [tostring(0/0)] = '0/0'}"
and used in my own table lib.

[Robin]

Yeah, someone else ran into the same issue, so I updated Ser a while ago (still need to do the same for Lady, hmm)

Nice job, Robin I like the lib a lot.

Thanks!

Hope you don't mind but I stole your line:
"local oddvals = {[tostring(1/0)] = '1/0', [tostring(-1/0)] = '-1/0', [tostring(0/0)] = '0/0'}"
and used in my own table lib.

It's open source for a reason.

[Positive07]

Nice lib, works really fast and the results are always right! My favorite serialization lib out there

[ivan]

Hey Robin,
I've been looking into serialization a little bit and playing with the Ser source code from Github.
Great work there with keeping the code succinct.
I have a couple of cosmetic proposals about the code.

Code: Select all

local function write_table_ex(t, memo, rev_memo, srefs, name)
	...
	m[mi > 3 and mi or mi + 1] = '}'
	return concat(m)
end

I believe we could just use concat(m, ',') instead of inserting every comma.

Code: Select all

  -- separate pairs with comma
	return '_[' .. name .. ']={' .. concat(m, ',') .. '}'
end

And I think I found an extra operation here:

Code: Select all

	elseif ty == 'boolean' or ty == 'nil' then
		t = tostring(t)
		return tostring(t)

Where the line t = tostring(t) could be omitted altogether.
My uglier, modified version is available in the following repo under "utils.table.ser":
https://bitbucket.org/itraykov/utils
I might make a few other changes in the future, if you don't mind.
Good work though, very nicely made library.

[Robin]

Thanks, ivan.

The second one I changed right away, it's a leftover from an older version. Good catch!

The second one, I'm reluctant about, because it might make the code slower. I try to avoid concatenation with .. as much as I can, especially if it's possible a large string might need to be copied, specifically after a call to table.concat. Maybe I need to run some benchmarks to see what the damage'll be.

[ivan]

Cool.

The second one, I'm reluctant about, because it might make the code slower. I try to avoid concatenation with .. as much as I can, especially if it's possible a large string might need to be copied, specifically after a call to table.concat. Maybe I need to run some benchmarks to see what the damage'll be.

I tested with a similar example to the one from Github, and it ran slightly (~5-10%) faster than the original Ser.
It might depend on the input though.
Yeah, I know what you mean about the .. operator, I tried a version with string.format instead of .. but it was slower.
.. creates intermediate strings which uses more stack/memory but it's still faster than the overhead of string.format (at least in this case).

Attached is the test I used.

[Robin]

I played around with it a bit, and it seems that your solution is an overall improvement, only suffering if a table contains very few, very long strings. I'll try to find a way around that and incorporate your improvements!

[ivan]

How's it going, Robin?
I've been using ser for a few months now and it's been great. No problems whatsoever.
However I had to make one very minor change:

Code: Select all

t = format("%.17g", t)

This line assumes that the user is running a Lua distribution with doubles.
Unfortunately this is not particularly true in my case (using Agen which has floats).
I suggest removing that line altogether.

Alternatively there are tricks you can try to determine the floating point precision
of numbers in Lua, but it's ugly and not recommended:

Code: Select all

-- number of significant bits
local bits = 53
for b = 16, 128 do
  local n = 2^b
  if n + 1 == n then
    bits = b
    break
  end
end
-- number of decimal digits precision
local digits = math.log10(2^bits)
-- plus one
digits = math.ceil(digits) + 1
-- format string
local formats = "%." .. digits .. "g"

print(formats)

-- single: 11 s.bits, 3.311 decimal digits precision
-- float: 24 s.bits, 6 to 9 decimal digits precision
-- double: 53 s.bits, 15 to 17 decimal digits precision
-- quad: 113 s.bits, 33 to 36 decimal digits precision 

Thanks again for the great lib and cheers!

[Robin]

Thanks for the heads-up, ivan. What happens with that line for a single precision build?

The problem with removing that line is that tostring() is not precise enough for double precision builds of Lua. I'll have to think about the best course of action here.