"Loosing" my thread?

[Germanunkol]

Hi,

I'm using love.thread to generate a map for my game. It works perfectly - mostly. However, some few times, I am loosing all connectivity to the thread. The thread object is not nil, I get no errors when indexing it to get new data, I just don't get any data.

First, I created a "print" function inside the thread -> all this one does is it sends packets back to the main thread which are then printed to console from the main thread. This works great for debugging, but not in the above case, where the thread seems to be "lost". In this case, I get nothing returned to the main thread.
Next, I tried logging to a file instead from inside the thread. After "loosing" the thread, I get the same output in the file as without loosing it, however, I do not know if this is from this time the thread was created or from the time before, when it ran fine.

This code creates and handles the thread:
As I said, mapGenerateThread does never become nil...

Code: Select all

function map.generate(width, height, seed, tutorialMap)
	if not mapGenerateThread then
	
		print("Generating map!")
	
		newMapStarting = true
		mapRenderPercent = nil
		mapGeneratePercent = nil
		numTrains = 0
		mapSeed = seed
	
		if not DEDICATED then
			console.init(love.graphics.getWidth(),love.graphics.getHeight()/2)
	
			love.graphics.translate(camX + love.graphics.getWidth()/(2*camZ), camY + love.graphics.getHeight()/(2*camZ))
			
			simulation.stop()
			
			loadingScreen.addSection("Generating Map")
		end
	
		mapImage,mapShadowImage,mapObjectImage = nil,nil,nil
		
		if not width then width = 4 end
		if not height then height = 4 end
		if not seed then seed = 1 end
		
		if width < 4 then
			print("Minimum width is 4!")
			width = 4
		end
		if height < 4 then
			print("Minimum height is 4!")
			height = 4
		end

	
		if tutorialMap then
			print("Generating Map...", tutorialMap.width, tutorialMap.height)
		else
			print("Generating Map...", width, height)
		end
		-- mapImage, mapShadowImage, mapObjectImage = map.render()
		mapGenerateThread = love.thread.newThread("mapGeneratingThread" .. mapGenerateThreadNumber, "Scripts/mapGenerate.lua")
		
		mapGenerateThreadNumber = mapGenerateThreadNumber + 1
		mapGenerateThread:start()
		if tutorialMap then mapGenerateThread:set("tutorialMap", TSerial.pack(tutorialMap)) end
		mapGenerateThread:set("width", width )
		mapGenerateThread:set("height", height )
		mapGenerateThread:set("seed", seed )
		
		mapGenerateStatusNum = 0
		
		mapGenerateMsgNumber = 0
		prevStr = nil
		print("mapGenerateThread", mapGenerateThread)
	else
	
		percent = mapGenerateThread:get("percentage")
		if percent and loadingScreen then
			loadingScreen.percentage("Generating Map", percent)
		end
		
		
		-- first, look for new messages:
		str = mapGenerateThread:get("msg" .. mapGenerateMsgNumber)
		while str do
			print(str, mapGenerateMsgNumber)
			if prevStr == str and str == "I'm done!" then
				print("same message twice")
				love.event.quit()
			end
			prevStr = str
			mapGenerateMsgNumber = incrementID(mapGenerateMsgNumber)
			str = mapGenerateThread:get("msg" .. mapGenerateMsgNumber)
		end
		
		-- then, check if there was an error:
		err = mapGenerateThread:get("error")
		if err then
			print("MAP GENERATING THREAD error: " .. err)
			love.event.quit()
		end
		
		
		status = mapGenerateThread:get("status" .. mapGenerateStatusNum)
		if status and loadingScreen then
			mapGenerateStatusNum = incrementID(mapGenerateStatusNum)
			loadingScreen.addSubSection("Generating Map", status)
		end
		
		status = mapGenerateThread:get("status")
		if status == "done" then
			print("Generating done!")
			
			curMap = TSerial.unpack(mapGenerateThread:demand("curMap"))
			curMapRailTypes = TSerial.unpack(mapGenerateThread:demand("curMapRailTypes"))
			curMapOccupiedTiles = TSerial.unpack(mapGenerateThread:demand("curMapOccupiedTiles"))
			curMapOccupiedExits = TSerial.unpack(mapGenerateThread:demand("curMapOccupiedExits"))
			
			if loadingScreen then
				loadingScreen.percentage("Generating Map", 100)
			end
			map.print("Finished Map:")
			mapGenerateThread = nil
			if not DEDICATED then
				map.render(curMap)
			else
				sendMap()		-- important! send before running the map!
				runMap()
			end
			collectgarbage("collect")
			
			return curMap
--		elseif status and loadingScreen then
--			loadingScreen.addSubSection("Generating Map", status)
		end
	end
end

And this is the thread itself:

Code: Select all

require("love.filesystem")

file = love.filesystem.newFile( "threadLog.txt" )
file:open("w")

file:write("1\n")
thisThread = love.thread.getThread()

package.path = "Scripts/?.lua;" .. package.path

require("mapUtils")
require("TSerial")
require("misc")

local msgNumber = 0
print = function(...)
	sendStr = ""
	for i = 1, #arg do
		if arg[i] then
			sendStr = sendStr .. arg[i] .. "\t"
		end
	end
	thisThread:set("msg" .. msgNumber, sendStr)
	msgNumber = incrementID(msgNumber)
end

print("2")
file:write("2\n")


print("4")
width = thisThread:demand("width")

file:write("5\n")
print("5")
height = thisThread:demand("height")
seed = thisThread:demand("seed")
tutorialMap = thisThread:get("tutorialMap")

print("6")
file:write("6\n")


if tutorialMap then
	tutorialMap = TSerial.unpack(tutorialMap)
	width = tutorialMap.width
	height = tutorialMap.height
end

print("7")
file:write("7\n")

math.randomseed(seed)
if not tutorialMap then curMap = {width=width, height=height, time=0} end
curMapOccupiedTiles = {}
curMapOccupiedExits = {}
curMapRailTypes = {}


print("8")
file:write("8\n")
thisThread:set("percentage", 0)

for i = 0,width+1 do
	if not tutorialMap then curMap[i] = {} end
	curMapRailTypes[i] = {}
	if i >= 1 and i <= width then 
		curMapOccupiedTiles[i] = {}
		curMapOccupiedExits[i] = {}
		for j = 1, height do
			curMapOccupiedTiles[i][j] = {}
			curMapOccupiedTiles[i][j].from = {}
			curMapOccupiedTiles[i][j].to = {}
		
			curMapOccupiedExits[i][j] = {}
		end
	end
end

print("9")
file:write("9\n")
if not tutorialMap then

	--thisThread:set("status", "rails")
	
	threadSendStatus( thisThread,"rails")
	thisThread:set("percentage", 10)
	generateRailRectangles()
	thisThread:set("percentage", 20)

	clearLargeJunctions()
	thisThread:set("percentage", 30)
	connectLooseEnds()
	thisThread:set("percentage", 40)
else
	curMap = tutorialMap
end

print("10")
file:write("10\n")
calculateRailTypes()
thisThread:set("percentage", 50)

if not tutorialMap then
	--thisThread:set("status", "houses")
	
	threadSendStatus( thisThread,"houses")
	placeHouses()
	thisThread:set("percentage", 60)

	--thisThread:set("status", "hotspots")
	
	threadSendStatus( thisThread,"hotspots")
	placeHotspots()
	thisThread:set("percentage", 70)
end

generateRailList()
thisThread:set("percentage", 90)

-- return the results to parent (main) thread:
thisThread:set("curMap", TSerial.pack(curMap))
thisThread:set("curMapRailTypes", TSerial.pack(curMapRailTypes))
thisThread:set("curMapOccupiedTiles", TSerial.pack(curMapOccupiedTiles))
thisThread:set("curMapOccupiedExits", TSerial.pack(curMapOccupiedExits))
thisThread:set("status", "done")

print("I'm done!")
file:write("Done\n")
return

Could anyone help me finding out what happens to my thread? Any ideas what else I could try to debug the problem?

This is running on the server, headless (no love.graphics initialized) and the server runs fine for a few hours (meaning it generates about 100 or 200 maps before all this thread-loosing happens.

[bartbes]

Does it error out, perhaps? (Check by reading the "error" message on the thread.)

[Germanunkol]

I do... this is part of the thread handling. That should do the job, right?

Code: Select all

 -- then, check if there was an error:
      err = mapGenerateThread:get("error")
      if err then
         print("MAP GENERATING THREAD error: " .. err)
         love.event.quit()
      end

However, the last thing I get on the console is:
"mapGenerateThread" Thread

which is the output of:
print("mapGenerateThread", mapGenerateThread)
which is called just after starting the thread and setting the values for it.
After this line, I get nothing else...it just stays at that for hours.
However, I do know that the function is called again (I checked by using debug printouts -> the "else" part, which is supposed to handle the thread, is called once every frame.)

[Germanunkol]

An update: When the problem occurs, it seems the thread doesn't even get to the first:
file:write("1\n")
Write instruction. I found out by emptying the log file before starting the thread... and it fills it, usually, but not when the problem occurs. So it seems the thread might not even be started correctly?

[kikito]

It's going to be simpler for everyone if you just show your .love .

[Germanunkol]

Hm... problem is that it really needs to run for multiple hours to get the error to occur. Not sure if anyone wants to do that.

It's also quite a large project, and I didn't want to ask anyone to debug it for me, I was hoping for hints on how I could go about debugging it myself.
However, if anyone wants to try, it's on github. Simply do:

Code: Select all

mkdir trAInsported
cd trAInsported
git clone https://github.com/Germanunkol/trAInsported.git
love . --server -c 10 -m 20

This will start in dedicated server mode, and generate a new map approimately every 10+20=30 seconds.
The map generation is started in the Scripts/map.lua file in the map.generate function which is called at the beginning of every new match and will be called every frame until the thread is done running.
It will generate the thread and then it'll keep calling the function, running the "else" part. But here, while the mapGenerateThread pointer is never set to nil, there's no response from the thread whatsoever.

notice that to remove the entire project again you'll need to remove the trAInsported folder AND the folder in .local/share/love/trAInsported -> depending on how the game is started, it might've generated some images and put them there (only when started in client mode).

[Germanunkol]

I noticed that the lower the -m parameter is (= Match time) the faster the problem seems to occur, and usually around match 240 or 250.
This looks to me like some memory overflow or something?
But I checked... after the problem occured, the process takes only 3% or my RAM (running on a raspberry-pi, so RAM is relatively low)

[Robin]

Hm... problem is that it really needs to run for multiple hours to get the error to occur.

Often, we don't have to run the .love to see the problem. It will still help us help you if you upload your .love.

[kikito]

Well, the code is on github, which is very easy to read, too.

I had the server running for a couple hours on my machine but did not find anything obviously wrong (got no error messages, the console kept printing stuff about trains and maps and passengers moving around).

I suspect there is some rare condition that makes the thread "exit". I remember having read somewhere that threads took some time to "die" after they had ended. It might have been related with garbage collection. But I don't remember the details.

[Germanunkol]

Thanks for trying!

Did you start the server with a short match and cooldown time? -m and -c, minimum is 10...

The last message I get is:
mapGenerateThread, ID Thread 249
Which is the output of:
print("mapGenerateThread, ID", mapGenerateThread, mapGenerateThreadNumber)
Interstingly, I just noticed that it always seems to be thread number 249 which stops working!
So it runs 248 matches and then stops.
This is only on my Pi though, I'll try again on my home PC when I get home. If it's RAM-related, it might be a different number on each PC.