Silly strings...

General discussion about LÖVE, Lua, game development, puns, and unicorns.
Post Reply
User avatar
Chief
Party member
Posts: 101
Joined: Fri Mar 12, 2010 7:57 am
Location: Norway, 67° north

Silly strings...

Post by Chief »

Since this isn't related to the LOVE part of lua, i just dropped it here.

I don't know much about the stuff that goes on "backstage" in LUA, but i noticed something odd the other day...

I were making this "table/nested tables to file function", which basically made the table into a very long string. When i finished the function, I wanted to test the limits - so i made a humongous table! After about 4-5 seconds LOVE crashed, which was odd. I figured that it had to be the string that was getting too long, since the writing process hadn't started yet.

The question to you guys: Does LUA have a "max signs in string limit"... or was it gremlins?

PS. the function was running in a coroutine, and adding each index in the table to a string for each turn!
User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

Re: Silly strings...

Post by Robin »

Can we see your code? I have a suspicion about the problem, which won't occur if you use table.concat, but usually does when concatenating strings in other ways.
Help us help you: attach a .love.
User avatar
Chief
Party member
Posts: 101
Joined: Fri Mar 12, 2010 7:57 am
Location: Norway, 67° north

Re: Silly strings...

Post by Chief »

The code is extremely braided and long, but i'll pick out the important parts:

Reading the table:

Code: Select all

-- Yada-yada lots of generation code

local str = "local map = {}\n"
		local function add( s )
			str = str .. s;
		end
		local num = 1

local function write(n, str)
			local path = "maps/" .. name .. "/"
			if love.filesystem.isDirectory( path ) then
				local success = love.filesystem.write( path .. tostring(n) .. ".lua", str )
				return success;
			else
				local ok = love.filesystem.mkdir( path )
				if ok then
					local success = love.filesystem.write( path .. tostring(n) .. ".lua", str )
					return success;
				else
					return false, "could not save table"
				end
			end
end

for x, t in pairs(map) do
		
			add( "map["..x.."] = {}\n" )
			
			for y, g in pairs(map[x]) do
			
				add( "map["..x.."]["..y.."] = {}\n" )
				
				add( table.getString( "map["..x.."]["..y.."]", g ) )
			end
			add( "return map;" )
			
			write(num, str)
			num = num + 1
			str = "local map = {}\n"
			percent = 25 + (x/#map)*75
			-- this is where i reran the thread.
		end
end
The table-to-string code:

Code: Select all

function table.getString( base, tab )
	local str = ""
	local function add( s )
		str = str .. s;
	end
	for id, info in pairs(tab) do
		local text = base
		if type(id) == "string" then
			text = text .. "['" .. id .. "'] = "
		elseif type(id) == "number" then
			text = text .. "[" .. id .. "] = "
		else
			text = text .. "." .. id .. " = "
		end
		
		if type(info) == "string" then
			add( text .. "'" .. tostring(info) .. "'" .. "\n" )
		elseif type(info) == "bool" or type(info) == "number" then
			add( text .. tostring(info) .. "\n" )
		elseif type(info) == "table" then
			add( text .. tostring(table.nestedTableString( info )) .. "\n" )
		else
			add( text .. tostring(info) .. "\n" )
		end
	end
	return str;
end

function table.nestedTableString( tab )
	local str = "{ "
	local function add( s )
		str = str .. s;
	end
	for k,v in pairs( tab ) do
		if type(v) == "string" then
			add( "'" .. tostring(v) .. "'" .. ", " )
		elseif type(v) == "bool" or type(v) == "number" then
			add( tostring(v) .. ", " )
		elseif type(v) == "table" then
			add( tostring(table.nestedTableString( v )) .. ", " )
		else
			add( tostring(v) .. ", " )
		end
	end
	
	add( "}" )
	
	return str
end
I "fixed" it by making a separate file for each index which contains another table...

The important part is that it crashes when the "str" is too long.

By the way - you may all use the table functions if you want to.
User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

Re: Silly strings...

Post by Robin »

It seems I was right.

The problem is, concatenating strings makes a new copy. Thus a = b .. c has three strings: b, c and b .. c. If you concatenate yet another string, you get a = b .. c .. d. Here you get five strings: b, c, d, b .. c and b .. c .. d. When you have one million strings in a table, you will need to create one million (minus one) new strings. What's worse, making strings doesn't take constant time: the longer the strings get, the more bytes need to be copied. So in the end, that explodes (usually because there is no memory left for the program to use).
Help us help you: attach a .love.
User avatar
Chief
Party member
Posts: 101
Joined: Fri Mar 12, 2010 7:57 am
Location: Norway, 67° north

Re: Silly strings...

Post by Chief »

Ah right! Thanks!
User avatar
kikito
Inner party member
Posts: 3153
Joined: Sat Oct 03, 2009 5:22 pm
Location: Madrid, Spain
Contact:

Re: Silly strings...

Post by kikito »

It's much better to use table.concat:

Code: Select all

a = a .. b
a = a .. c
a = a .. d
return a
This creates 7 strings in memory: the initial a, b, c & d, plus a .. b, a .. c and finally a .. d. It gets worse with every bit added to a.

Using table.concat, you only create a,b,c,d & the final concatenation, plus a table:

Code: Select all

local buffer = {}
table.insert(buffer, a)
table.insert(buffer, b)
table.insert(buffer, c)
table.insert(buffer, d)
return table.concat(buffer)
With a big number of strings, tables perform much better, memory-wise.
When I write def I mean function.
User avatar
tentus
Inner party member
Posts: 1060
Joined: Sun Oct 31, 2010 7:56 pm
Location: Appalachia
Contact:

Re: Silly strings...

Post by tentus »

Robin wrote:It seems I was right.

The problem is, concatenating strings makes a new copy. Thus a = b .. c has three strings: b, c and b .. c. If you concatenate yet another string, you get a = b .. c .. d. Here you get five strings: b, c, d, b .. c and b .. c .. d. When you have one million strings in a table, you will need to create one million (minus one) new strings. What's worse, making strings doesn't take constant time: the longer the strings get, the more bytes need to be copied. So in the end, that explodes (usually because there is no memory left for the program to use).
Whuh... WHY would it be set up this way? Whats the advantage?
Kurosuke needs beta testers
User avatar
bartbes
Sex machine
Posts: 4946
Joined: Fri Aug 29, 2008 10:35 am
Location: The Netherlands
Contact:

Re: Silly strings...

Post by bartbes »

Advantage? That's just the way it is done. Note that this is because of the way C strings work.

EDIT: Oh right, as kikito's link says, the string objects stay around until the next garbage collection cycle, so yeah, that too.
As this is not a single operation, a .. b .. c is actually a .. (b .. c) (the order I don't know exactly, but you get the point), and each has a result that is then discarded and then there's a wait for the garbage collector, it takes a lot of memory for a short period of time. Due to the way the GC works, in such a loop, it probably won't kick in, or at least not often enough.
User avatar
kikito
Inner party member
Posts: 3153
Joined: Sat Oct 03, 2009 5:22 pm
Location: Madrid, Spain
Contact:

Re: Silly strings...

Post by kikito »

I think this explains the issue better than I could:

http://www.lua.org/pil/11.6.html
When I write def I mean function.
User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

Re: Silly strings...

Post by Robin »

bartbes wrote:As this is not a single operation, a .. b .. c is actually a .. (b .. c) (the order I don't know exactly, but you get the point)
You're right.
Lua 5.1 Reference Manual wrote:The concatenation ('..') and exponentiation ('^') operators are right associative.
Which actually makes it slightly better for repeatedly adding short strings to long strings, but it's still awful of course.
tentus wrote:Whuh... WHY would it be set up this way? Whats the advantage?
Thing is, the options are really this or mutable strings. And mutable strings just don't work so well in languages like Lua, since it probably means you'll be changing the same string in other places unexpectedly as well.
Help us help you: attach a .love.
Post Reply

Who is online

Users browsing this forum: No registered users and 57 guests