## Prevent freezing while working with large files?

Questions about the LÖVE API, installing LÖVE and other support related questions go here.
Forum rules
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

### Prevent freezing while working with large files?

Hey, I'm writing some dev tools for a future game. Currently I need to read approximately 200 text files line by line which have a size of 2 KB up to 7 MB. Everything's working fine, I just wonder if there's a way to prevent the ugly freezing while reading the files. I'm using imgui as an interface and it would be nice to make use of a progress bar while reading those files. I've tried love.thread and also coroutines but my knowledge stops there and I have no idea what to do

Code: Select all

function love.draw()
--drawing stuff
if Working == false then
Working = true
end
end
--drawing stuff
end

while Working do
Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
end
end
Working = false
end
end


KayleMaster
Party member
Posts: 212
Joined: Mon Aug 29, 2016 8:51 am

### Re: Prevent freezing while working with large files?

Threads are the way to go, but it's a bit complicated. I'm on mobile now but maybe I can help later.

KayleMaster
Party member
Posts: 212
Joined: Mon Aug 29, 2016 8:51 am

### Re: Prevent freezing while working with large files?

Ok, here the zip file. I didn't make a .love but you can run this the normal way.
Also side note, the conf file is for 0.10.2 so if you're on 11 go ahead and copy the conf from here:
https://love2d.org/wiki/Config_Files
And change console = true (if you're on windows), and vsync = true just to limit the fps the easy way.

So now I'm gonna go through the important part of my code step by step:

Code: Select all

local thread -- Our thread object.

end

We create local variables for our thread and two of our channels. The sendLine channel stored in receiveLine will be used to receive the lines read from the files from the thread. The action thread is used to notify when the thread is over, although it really isn't necessary since we can just use thread:isRunning().

Code: Select all

function love.update(dt)
imgui.NewFrame()

-- Make sure no errors occured.
assert( not error, error )

if Working then
if threadIsDone then Working = false end
end
end

miniTimer = miniTimer + 1
if miniTimer == 20 then
freezeNumber = freezeNumber + 1
freezeNumber = freezeNumber % 100
end
miniTimer = miniTimer % 30
end

The two lines below the Make sure no errors occured are very important. They will notify you if something went wrong in the thread. Without those you wouldn't know. When I think about it, that code can be put in the 'if Working' block, since if it's not Working the thread wouldn't be running as well.

Code: Select all

local threadIsDone = threadAction:pop() or not thread:isRunning()
if threadIsDone then Working = false end

This is self explanatory. The only change I would make was add an else and put the rest of the block inside it. But it doesn't matter.

Code: Select all

local received = receiveLine:pop()
end

We pop from the receiveLine channel (a channel is essentially a queue, a first in - first out data structure). If there's nothing, we'll have nil in received so we check for that.
The miniTimer code is to visualise if the game freezes while reading.

Code: Select all

function love.draw()
--drawing stuff
if Working == false then
Working = true
end
end

imgui.Render()
love.graphics.print(freezeNumber)
--drawing stuff
end

We start the thread using the method :start. You can find the methods for channels and threads here:
Channel

Code: Select all

require 'love.filesystem'

local channel = {}
local Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
channel.sendLine:supply(Line)
end
end
channel.action:push(true)

We load the love.filesystem module because on a thread, only the love.thread module is loaded and if we need any love modules we can just require them like so. This time I decided to store the channels in a table, I find it more readable this way.
I use the :supply method to send the Line over to the main thread. The other method is push, but there's a catch - if you use push and the main thread is busy, the Lines will fill up the queue and use a lot of memory. This memory won't be freed and considering you're reading lots of files I figured supply would be better. What supply does is it sends a message to a thread Channel and wait for a thread to accept it as opposed to the push which just sends the message and continues the operation. Using push in this case won't freeze the thread waiting for the main thread to accept the message but may use a bit more memory.
At the end I send a simple boolean through the action channel to notify we're done. As I said earlier this isn't necessary since we can just use thread:isRunning() but I did it to show that you can have many channels in a thread.

Attachments

typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

### Re: Prevent freezing while working with large files?

Oh wow, thanks a lot for your effort. I'm not home until tomorrow, so I can't give it a try earlier.

KayleMaster
Party member
Posts: 212
Joined: Mon Aug 29, 2016 8:51 am

### Re: Prevent freezing while working with large files?

You're welcome. One more side note - I dunno if the thread dies properly, but I just read that the thread is killed when it returns. So just add return true or something at the end of the thread code in threadRead.

typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

### Re: Prevent freezing while working with large files?

So, I guess I got the most stuff now, but can you tell me how to send arguments to the thread? I know i can puth them into thread:start(Args) but I wasn't able to find out how to receive them inside the thread code. I tried making more channels, using channel:push() and channel:pop() which kind of worked, but it brought my computer to it's knees, nearly freezing windows for some minutes

Nixola
Inner party member
Posts: 1940
Joined: Tue Dec 06, 2011 7:11 pm
Location: Italy

### Re: Prevent freezing while working with large files?

The arguments will be received as vararg. For example:
main.lua:

Code: Select all

thread = love.thread.newThread("thread.lua")


Code: Select all

local user, pass, port = ...
print(user, pass, port) -- this will print foo     bar     4200
lf = love.filesystem
ls = love.sound
la = love.audio
lp = love.physics
li = love.image
lg = love.graphics

KayleMaster
Party member
Posts: 212
Joined: Mon Aug 29, 2016 8:51 am

### Re: Prevent freezing while working with large files?

I knew I missed something!

If you're unsure how many arguments you're gonna send, you can also do:

Code: Select all

local arg_table = {...}
print(arg_table[1], arg_table[2])


typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

### Re: Prevent freezing while working with large files?

Okay, everything is working fine now expect some new thing i wanted to build in

Since the whole process of reading ~50MB of Textfiles line by line is super slow, I've tried to make use of a cancel button. I've tried to send an argument to the thread which should cancel the whole filereading. That's what I've tried:

Code: Select all

--main.lua & thread.lua
DoCancel = false

--main.lua
function love.draw()
--draw stuff
if imgui.Button("Cancel") then
DoCancel = true
ChannelCancel:push(DoCancel)
end
end

Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
DoCancel = ChannelCancel:pop()
if DoCancel then break end
end
end

Sadly, it's not working at all. When using print(DoCancel) its always false (sometimes even nil at first run with some earlier attempts). I've tried to put the push command into love.update() but that slowed my computer totally down.

I guess that's the last hurdle with all this new threading

KayleMaster
Party member
Posts: 212
Joined: Mon Aug 29, 2016 8:51 am

### Re: Prevent freezing while working with large files?

No, no, no. Why would you put :push into love.update. That would fill up the queue with the same messages 60 messages a second!
Anyways, you'd need to return, not break. Since you have 2 loops you only break from the outer one, the inner one is still working.
If you still have to do some stuff after cancel and return wouldn't work for you, you can also do:

Code: Select all

Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
DoCancel = ChannelCancel:pop()
if DoCancel then goto done end
end
end
::done::
--do more stuff here if you want

If you want to do it with breaks, you can do it like so: (I think)

Code: Select all

Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
DoCancel = ChannelCancel:pop()
if DoCancel then break end