Prevent freezing while working with large files?

Questions about the LÖVE API, installing LÖVE and other support related questions go here.
Forum rules
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Thanks alot, I really appreciate your help! Everything is working now. Sadly, it seems like Love isn't the right environment for this kind of tool. It's just super slow and after some minutes, it's freezing my computer At least I've learned a lot for future projects

KayleMaster
Party member
Posts: 197
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Can you share the .love file? I'm curious as to why it's slow because with luajit it's only 1.3x times slower than C.

typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

I can send it later when I'm at home. Imho it's the iterating of some of the files. The largest ones have up to 600k lines and it takes really long to read it line by line. I've tried getting the file as a filedata object and reading char by char until "\n" to get the lines but it's as slow as the other method.

KayleMaster
Party member
Posts: 197
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Well in that case, yeah. How much faster do you need ? It might be better to read the whole file in the thread without sending each line to the other thread and do the stuff you need to do with it there.

typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

That's how i did build it allready. The whole reading, analyzing, manipulating and writing stuff is run in the thread now. Problem is, the progressbar stuff is drawn in main so somehow I either need to send the lines or the progress calculation to main which both causes massive lags and freezes. Well, it's way better when using channel:supply() but it's taking hours then

I did build an earlier version of such a tool reading, manipulating and writing the same files in VB.net some months ago and everything was done in seconds. Here, a single medium-sized file alone takes longer

zorg
Party member
Posts: 2502
Joined: Thu Dec 13, 2012 2:55 pm
Location: Absurdistan, Hungary
Contact:

Re: Prevent freezing while working with large files?

I still don't really get the issue to be honest; reading in from disk/ssd into ram line by line will always be slow as heck (doing it byte-by-byte is worse), just load it in all at once, or use a large enough buffer, like in the megabytes range; updating your progress bar from 0 to 100% in one step is not any less correct than sacrificing loading speed deliberately just so your users could look at a line being filled

Also, if you have to process the data further after you loaded it in, you can still do that, in RAM, which will be many times faster than the loadline-process loop.

By the way, you never really explained why you needed to read the file line-by-line; and "to have the progress bar graphic update neatly" is not a good reason ever.
Me and my stuff True Neutral Aspirant. Why, yes, i do indeed enjoy sarcastically correcting others when they make the most blatant of spelling mistakes. No bullying or trolling the innocent tho.

typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

I've never said I need to read it line by line, also as I wrote earlier today, I've also tried working with the whole file as FileData which didn't boost the performance. Afaik, love.filesystem.lines and love.filesystem.read are the only options, according to the wiki.

Also, I don't do a 0 to 100 thing at my progressbar. It's 100 / NoOfFiles, added after every file so users can see what's happening. It doesn't have to be "shiny" but usable.

KayleMaster
Party member
Posts: 197
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

It sounds like you can't send your progress properly.

Code: Select all

--thread.lua
Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
DoCancel = ChannelCancel:pop()
if DoCancel then goto done end
channel.progress:pop()
channel.progress:push(progressNumber)
end
end
::done::

It would not cause freezes if we only kept one message in the queue. So we only pop/push from the thread. We do :peek from the main thread instead, which doesn't pop the message.

Code: Select all

--main.lua or wherever
local progressNumber = channelProgress:peek()

If even this causes slowdown, try sending the progress every 100 lines or so:

Code: Select all

local counter = 0
Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
counter = counter + 1
DoCancel = ChannelCancel:pop()
if DoCancel then goto done end
if counter == 100 then
channel.progress:pop()
channel.progress:push(progressNumber)
counter = 0
end
end
end
::done::

EDIT: nevermind, just read you need the file count
Try this:

Code: Select all

Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
channel.progress:pop()
channel.progress:push(iFiles/#Files)
for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
DoCancel = ChannelCancel:pop()
if DoCancel then goto done end
end
end
::done::

And then just do channelProgress:peek() from the main.lua
Obviously you need to make a channel for progress as well.

pgimeno
Party member
Posts: 1495
Joined: Sun Oct 18, 2015 2:58 pm

Re: Prevent freezing while working with large files?

Strings in LuaJIT are slow to handle. Using string.byte tends to give much better performance than string.sub. Not sure if that's the issue here, but I thought it was worth noting.
Thrust II Reloaded - GifLoad for Löve - GSpöt GUI - My NotABug.org repositories - portland (mobile orientation)
The MS-Github repositories I had have been closed after the acquisition announcement and will be removed in the near future.

typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

I guess the issue is working with the file itself. When using other languages, I just load the whole file into an array and save it after work is done. Guess using FileData = love.filesystem.read(File) doesn't do the trick with lua/love since it's the same speed als love.filesystem.lines

Who is online

Users browsing this forum: Exabot [Bot], pgimeno and 9 guests