Prevent freezing while working with large files?

Questions about the LÖVE API, installing LÖVE and other support related questions go here.
Forum rules
Before you make a thread asking for help, read this.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Prevent freezing while working with large files?

Post by typx » Sat Jun 16, 2018 10:16 pm

Hey, I'm writing some dev tools for a future game. Currently I need to read approximately 200 text files line by line which have a size of 2 KB up to 7 MB. Everything's working fine, I just wonder if there's a way to prevent the ugly freezing while reading the files. I'm using imgui as an interface and it would be nice to make use of a progress bar while reading those files. I've tried love.thread and also coroutines but my knowledge stops there and I have no idea what to do :D

Code: Select all

function love.draw()
  --drawing stuff
  if imgui.Button("Read files") then
    if Working == false then
      Working = true
      ReadFiles()
    end
  end
  --drawing stuff
end

function ReadFiles()
  while Working do
    Files = love.filesystem.getDirectoryItems("data")
    for iFiles = 1, #Files do
      for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
        --read lines
      end
    end
    Working = false
  end
end

KayleMaster
Party member
Posts: 212
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Post by KayleMaster » Sun Jun 17, 2018 1:36 am

Threads are the way to go, but it's a bit complicated. I'm on mobile now but maybe I can help later.

KayleMaster
Party member
Posts: 212
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Post by KayleMaster » Sun Jun 17, 2018 10:20 am

Ok, here the zip file. I didn't make a .love but you can run this the normal way.
Also side note, the conf file is for 0.10.2 so if you're on 11 go ahead and copy the conf from here:
https://love2d.org/wiki/Config_Files
And change console = true (if you're on windows), and vsync = true just to limit the fps the easy way.

So now I'm gonna go through the important part of my code step by step:

Code: Select all

local thread -- Our thread object.
local receiveLine = love.thread.getChannel( "sendLine" )
local threadAction = love.thread.getChannel( "action" )


function love.load()
    thread = love.thread.newThread( "threadRead.lua" )
end
We create local variables for our thread and two of our channels. The sendLine channel stored in receiveLine will be used to receive the lines read from the files from the thread. The action thread is used to notify when the thread is over, although it really isn't necessary since we can just use thread:isRunning().

We load the thread code using newThread and the directory of the lua file.

Code: Select all

function love.update(dt)
    imgui.NewFrame()
    
    -- Make sure no errors occured.
    local error = thread:getError()
    assert( not error, error )

    if Working then
        local threadIsDone = threadAction:pop() or not thread:isRunning()
        if threadIsDone then Working = false end
        local received = receiveLine:pop()
        if received then
            print(received)
        end
    end

    miniTimer = miniTimer + 1
    if miniTimer == 20 then
        freezeNumber = freezeNumber + 1
        freezeNumber = freezeNumber % 100
    end
    miniTimer = miniTimer % 30
end
The two lines below the Make sure no errors occured are very important. They will notify you if something went wrong in the thread. Without those you wouldn't know. When I think about it, that code can be put in the 'if Working' block, since if it's not Working the thread wouldn't be running as well.

Code: Select all

local threadIsDone = threadAction:pop() or not thread:isRunning()
        if threadIsDone then Working = false end
This is self explanatory. The only change I would make was add an else and put the rest of the block inside it. But it doesn't matter.

Code: Select all

local received = receiveLine:pop()
        if received then
            print(received)
        end
We pop from the receiveLine channel (a channel is essentially a queue, a first in - first out data structure). If there's nothing, we'll have nil in received so we check for that.
The miniTimer code is to visualise if the game freezes while reading.

Code: Select all

function love.draw()
  --drawing stuff
  if imgui.Button("Read files") then
    if Working == false then
      Working = true
      thread:start()
    end
  end
  
  imgui.Render()
  love.graphics.print(freezeNumber)
  --drawing stuff
end
We start the thread using the method :start. You can find the methods for channels and threads here:
Channel
Thread

threadRead.lua

Code: Select all

require 'love.filesystem'

local channel = {}
channel.sendLine = love.thread.getChannel( "sendLine" )
channel.action = love.thread.getChannel( "action" )
print("Started reading")
local Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
    for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    --read lines
    channel.sendLine:supply(Line)
    end
end
channel.action:push(true)
We load the love.filesystem module because on a thread, only the love.thread module is loaded and if we need any love modules we can just require them like so. This time I decided to store the channels in a table, I find it more readable this way.
I use the :supply method to send the Line over to the main thread. The other method is push, but there's a catch - if you use push and the main thread is busy, the Lines will fill up the queue and use a lot of memory. This memory won't be freed and considering you're reading lots of files I figured supply would be better. What supply does is it sends a message to a thread Channel and wait for a thread to accept it as opposed to the push which just sends the message and continues the operation. Using push in this case won't freeze the thread waiting for the main thread to accept the message but may use a bit more memory.
At the end I send a simple boolean through the action channel to notify we're done. As I said earlier this isn't necessary since we can just use thread:isRunning() but I did it to show that you can have many channels in a thread.

Hope this helped, I'm pretty bad at writing tutorials but this should help you a little.
Attachments
ThreadExample.zip
(12.89 KiB) Downloaded 55 times

typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx » Sun Jun 17, 2018 10:36 am

Oh wow, thanks a lot for your effort. I'm not home until tomorrow, so I can't give it a try earlier.

KayleMaster
Party member
Posts: 212
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Post by KayleMaster » Sun Jun 17, 2018 11:01 am

You're welcome. One more side note - I dunno if the thread dies properly, but I just read that the thread is killed when it returns. So just add return true or something at the end of the thread code in threadRead.

typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx » Mon Jun 18, 2018 1:34 pm

So, I guess I got the most stuff now, but can you tell me how to send arguments to the thread? I know i can puth them into thread:start(Args) but I wasn't able to find out how to receive them inside the thread code. I tried making more channels, using channel:push() and channel:pop() which kind of worked, but it brought my computer to it's knees, nearly freezing windows for some minutes :D

User avatar
Nixola
Inner party member
Posts: 1940
Joined: Tue Dec 06, 2011 7:11 pm
Location: Italy

Re: Prevent freezing while working with large files?

Post by Nixola » Mon Jun 18, 2018 4:03 pm

The arguments will be received as vararg. For example:
main.lua:

Code: Select all

thread = love.thread.newThread("thread.lua")
thread:start("foo", "bar", 4200)
thread.lua:

Code: Select all

local user, pass, port = ...
print(user, pass, port) -- this will print foo     bar     4200
lf = love.filesystem
ls = love.sound
la = love.audio
lp = love.physics
lt = love.thread
li = love.image
lg = love.graphics

KayleMaster
Party member
Posts: 212
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Post by KayleMaster » Mon Jun 18, 2018 6:38 pm

I knew I missed something!

If you're unsure how many arguments you're gonna send, you can also do:

Code: Select all

local arg_table = {...}
print(arg_table[1], arg_table[2])

typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx » Tue Jun 19, 2018 1:54 pm

Okay, everything is working fine now expect some new thing i wanted to build in :D

Since the whole process of reading ~50MB of Textfiles line by line is super slow, I've tried to make use of a cancel button. I've tried to send an argument to the thread which should cancel the whole filereading. That's what I've tried:

Code: Select all

--main.lua & thread.lua
ChannelCancel = love.thread.getChannel("cancel")
DoCancel = false

--main.lua
function love.draw()
  --draw stuff
  if imgui.Button("Cancel") then
    DoCancel = true
    ChannelCancel:push(DoCancel)
  end
end

--thread.lua
Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    DoCancel = ChannelCancel:pop()
    if DoCancel then break end
    --read lines etc
  end
end
Sadly, it's not working at all. When using print(DoCancel) its always false (sometimes even nil at first run with some earlier attempts). I've tried to put the push command into love.update() but that slowed my computer totally down.

I guess that's the last hurdle with all this new threading :D

KayleMaster
Party member
Posts: 212
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Post by KayleMaster » Tue Jun 19, 2018 2:44 pm

No, no, no. Why would you put :push into love.update. That would fill up the queue with the same messages 60 messages a second!
Anyways, you'd need to return, not break. Since you have 2 loops you only break from the outer one, the inner one is still working.
If you still have to do some stuff after cancel and return wouldn't work for you, you can also do:

Code: Select all

Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    DoCancel = ChannelCancel:pop()
    if DoCancel then goto done end
    --read lines etc
  end
end
::done::
--do more stuff here if you want 
If you want to do it with breaks, you can do it like so: (I think)

Code: Select all

Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    DoCancel = ChannelCancel:pop()
    if DoCancel then break end
    --read lines etc
  end  
  if DoCancel then break end
end

EDIT: the reason why pushing every tick worked is because it just skipped the inner loop. This is unefficient because you push messages and the inner loop removes them (pop). You could get away with doing :peek instead. But again, you'll still be going through the outer loop.

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests