Prevent freezing while working with large files?

[KayleMaster]

You can still do that in C, using FFI, if you absolutely need to. I don't think loading a 600k line file into a contiguous array would work well tho.

[typx]

Well, at least in VB.net I do kinda the same, using the same files to read.

Code: Select all

Dim DataFiles() As String = IO.Directory.GetFiles("data", "*.sav")
For iFiles = 0 To DataFiles.Length - 1
  Dim Lines() As String = IO.File.ReadAllLines(DataFiles(iFiles))
  For iLines = 0 To Lines.Length - 1
    Line = Lines(iLines)
  Next
Next

This iterates all files in 4 seconds, while the love attempt takes several minutes for 1MB+ files, even without any graphical extras.

[KayleMaster]

I mean, I can optimize it if you need to. But I'd need your project folder and data to do that.
Anyways, Love2D isn't really meant for this, obviously it's a gamedev framework, but I don't think it's that slow.

[typx]

Yeah, I know it's not meant for this. But I try to solve tasks I think are relatively simple with lua/love to get used to the framework, imgui design and lua in general.
A friend of mine is working on a machine learning engine and I will provide dev tools and ui frontend for a game using it later (or maybe someday ), so I try to do as much in lua/love as possible, even if it's a bit out of the ordinary.

Sadly, I can't post the data files here, cause there's some licensing involved (they're from a game we're modding) but I could PM you if you're interested.

[grump]

This iterates all files in 4 seconds, while the love attempt takes several minutes for 1MB+ files, even without any graphical extras.

You must be running this on a potato or you're doing something severely wrong.

Code: Select all

local start = love.timer.getTime()

local lines = {}
for i = 1, 600000 do lines[i] = ('x'):rep(100) end
love.filesystem.write('test.txt', table.concat(lines, '\n'))

lines = {}
for line in love.filesystem.lines('test.txt') do
	lines[#lines + 1] = line
end

print(#lines, "lines:", love.timer.getTime() - start, "seconds")

Output:

Code: Select all

600000	lines:	0.8547670400003	seconds

Generating a table of 600k lines with 100 chars each, joining them, writing them to a file, reading them back in and put them in a table takes less than one second for me. That's 57 MB of data. Granted, the data is still in the filesystem cache when reading it back.

Are you using LÖVE 11? 0.10.2 and below had a bug in love.filesystem.lines that made it perform extremely poorly when reading from a love file.

[typx]

Ah, yeah I'm using 0.10.2, meh
Will give it a try with 11.1 tomorrow.

[typx]

Updating love helped a lot, reading just takes several seconds now Sadly, when I try to manipulate and rewrite the files now, it's getting super slow again love.filesystem.append single lines and also love.filesystem.write a huge string variable are unusable. I know writing to a table and just saving as lua is the way to go, but since we're modding an existing game which expects textfiles, that's not an option.

[bartbes]

Can't you just get an actual file handle in append mode and keep appending that way? Sure sounds more efficient than repeatedly opening and closing the same file.

[typx]

Oh yeah, that did speed up everything a lot. Thanks to all of you