Prevent freezing while working with large files?

Questions about the LÖVE API, installing LÖVE and other support related questions go here.
Forum rules
Before you make a thread asking for help, read this.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx »

Thanks alot, I really appreciate your help! Everything is working now. Sadly, it seems like Love isn't the right environment for this kind of tool. It's just super slow and after some minutes, it's freezing my computer :| At least I've learned a lot for future projects :)
KayleMaster
Party member
Posts: 234
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Post by KayleMaster »

Can you share the .love file? I'm curious as to why it's slow because with luajit it's only 1.3x times slower than C.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx »

I can send it later when I'm at home. Imho it's the iterating of some of the files. The largest ones have up to 600k lines and it takes really long to read it line by line. I've tried getting the file as a filedata object and reading char by char until "\n" to get the lines but it's as slow as the other method.
KayleMaster
Party member
Posts: 234
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Post by KayleMaster »

Well in that case, yeah. How much faster do you need ? It might be better to read the whole file in the thread without sending each line to the other thread and do the stuff you need to do with it there.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx »

That's how i did build it allready. The whole reading, analyzing, manipulating and writing stuff is run in the thread now. Problem is, the progressbar stuff is drawn in main so somehow I either need to send the lines or the progress calculation to main which both causes massive lags and freezes. Well, it's way better when using channel:supply() but it's taking hours then :D

I did build an earlier version of such a tool reading, manipulating and writing the same files in VB.net some months ago and everything was done in seconds. Here, a single medium-sized file alone takes longer :|
User avatar
zorg
Party member
Posts: 3436
Joined: Thu Dec 13, 2012 2:55 pm
Location: Absurdistan, Hungary
Contact:

Re: Prevent freezing while working with large files?

Post by zorg »

I still don't really get the issue to be honest; reading in from disk/ssd into ram line by line will always be slow as heck (doing it byte-by-byte is worse), just load it in all at once, or use a large enough buffer, like in the megabytes range; updating your progress bar from 0 to 100% in one step is not any less correct than sacrificing loading speed deliberately just so your users could look at a line being filled :P

Also, if you have to process the data further after you loaded it in, you can still do that, in RAM, which will be many times faster than the loadline-process loop.

By the way, you never really explained why you needed to read the file line-by-line; and "to have the progress bar graphic update neatly" is not a good reason ever.
Me and my stuff :3True Neutral Aspirant. Why, yes, i do indeed enjoy sarcastically correcting others when they make the most blatant of spelling mistakes. No bullying or trolling the innocent tho.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx »

I've never said I need to read it line by line, also as I wrote earlier today, I've also tried working with the whole file as FileData which didn't boost the performance. Afaik, love.filesystem.lines and love.filesystem.read are the only options, according to the wiki.

Also, I don't do a 0 to 100 thing at my progressbar. It's 100 / NoOfFiles, added after every file so users can see what's happening. It doesn't have to be "shiny" but usable.
KayleMaster
Party member
Posts: 234
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Post by KayleMaster »

It sounds like you can't send your progress properly.
EDIT: READ LAST CODE SNIPPET

Code: Select all

--thread.lua
Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    DoCancel = ChannelCancel:pop()
    if DoCancel then goto done end
    channel.progress:pop()
    channel.progress:push(progressNumber)
    --read lines etc
  end
end
::done::
It would not cause freezes if we only kept one message in the queue. So we only pop/push from the thread. We do :peek from the main thread instead, which doesn't pop the message.

Code: Select all

--main.lua or wherever
local progressNumber = channelProgress:peek()
If even this causes slowdown, try sending the progress every 100 lines or so:

Code: Select all

local counter = 0
Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    counter = counter + 1
    DoCancel = ChannelCancel:pop()
    if DoCancel then goto done end
    if counter == 100 then
    channel.progress:pop()
    channel.progress:push(progressNumber)
    counter = 0
    end
    --read lines etc
  end
end
::done::
EDIT: nevermind, just read you need the file count
Try this:

Code: Select all

Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do  
  channel.progress:pop()
  channel.progress:push(iFiles/#Files)
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    DoCancel = ChannelCancel:pop()
    if DoCancel then goto done end
    --read lines etc
  end
end
::done::
And then just do channelProgress:peek() from the main.lua
Obviously you need to make a channel for progress as well.
User avatar
pgimeno
Party member
Posts: 3544
Joined: Sun Oct 18, 2015 2:58 pm

Re: Prevent freezing while working with large files?

Post by pgimeno »

Strings in LuaJIT are slow to handle. Using string.byte tends to give much better performance than string.sub. Not sure if that's the issue here, but I thought it was worth noting.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx »

I guess the issue is working with the file itself. When using other languages, I just load the whole file into an array and save it after work is done. Guess using FileData = love.filesystem.read(File) doesn't do the trick with lua/love since it's the same speed als love.filesystem.lines
Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot], Bing [Bot] and 47 guests