Wrapper / abstraction layer mechanism that download needed files on demand

Questions about the LÖVE API, installing LÖVE and other support related questions go here.
Forum rules
Before you make a thread asking for help, read this.
glitchapp
Party member
Posts: 235
Joined: Tue Oct 05, 2021 10:34 am
Contact:

Wrapper / abstraction layer mechanism that download needed files on demand

Post by glitchapp »

Hello, I'm searching for a solution for big projects like some of the one I created to download the files on demand as the player progress through the game.

Someone advised me to implement an abstraction layer / wrapper to accomplish this, I have however no idea where to start and wonder if anyone has attempted that in love2d.

I'll be happy to receive feedback regarding this problem.

Thanks!
Xugro
Party member
Posts: 110
Joined: Wed Sep 29, 2010 8:14 pm

Re: Wrapper / abstraction layer mechanism that download needed files on demand

Post by Xugro »

The concept you are searching for is called lazy loading. Here is a sketch in pseudo code:

Code: Select all

function readFile(filename)
    if love.filesystem.getInfo(filename) == nil then
        -- download file
    end

    return love.filesystem.read(filename)
end
This will check if the file is already downloaded. If not it will download the file. Afterwards the file is read.
MrFariator
Party member
Posts: 509
Joined: Wed Oct 05, 2016 11:53 am

Re: Wrapper / abstraction layer mechanism that download needed files on demand

Post by MrFariator »

Of note, you may also want to do any such lazy loading on a separate thread, instead of blocking the main thread for N amount of time due to the downloads.
glitchapp
Party member
Posts: 235
Joined: Tue Oct 05, 2021 10:34 am
Contact:

Re: Wrapper / abstraction layer mechanism that download needed files on demand

Post by glitchapp »

Xugro wrote: Sat Jan 21, 2023 10:14 pm The concept you are searching for is called lazy loading. Here is a sketch in pseudo code:

Code: Select all

function readFile(filename)
    if love.filesystem.getInfo(filename) == nil then
        -- download file
    end

    return love.filesystem.read(filename)
end
This will check if the file is already downloaded. If not it will download the file. Afterwards the file is read.
Thanks, I didn't expect it to be that simple. I wonder if you can use that function to download the files from a git repository or it needs to be an html server.

Thanks for the info!
Last edited by glitchapp on Mon Jan 23, 2023 10:35 am, edited 1 time in total.
glitchapp
Party member
Posts: 235
Joined: Tue Oct 05, 2021 10:34 am
Contact:

Re: Wrapper / abstraction layer mechanism that download needed files on demand

Post by glitchapp »

MrFariator wrote: Sat Jan 21, 2023 10:41 pm Of note, you may also want to do any such lazy loading on a separate thread, instead of blocking the main thread for N amount of time due to the downloads.
That's a good idea but I will be happy if I manage to implement it no matter if it stops the main thread. Once done I will see how can I improve it and maybe attempt to add the loading process to a secondary thread.
User avatar
dusoft
Party member
Posts: 482
Joined: Fri Nov 08, 2013 12:07 am
Location: Europe usually
Contact:

Re: Wrapper / abstraction layer mechanism that download needed files on demand

Post by dusoft »

glitchapp wrote: Mon Jan 23, 2023 9:20 am Thanks, I didn't expect it to be that simple. I wonder if you can use that function to download the files from a git repository or it needs to be an html server.

Thanks for the info!
Surely you can download from anywhere, however for Git repo you would like to have some kind of API access to find and fetch the latest version.
Last edited by dusoft on Sun Jan 29, 2023 7:21 pm, edited 1 time in total.
Andlac028
Party member
Posts: 174
Joined: Fri Dec 14, 2018 2:27 pm
Location: Slovakia

Re: Wrapper / abstraction layer mechanism that download needed files on demand

Post by Andlac028 »

glitchapp wrote: Mon Jan 23, 2023 9:20 am Thanks, I didn't expect it to be that simple. I wonder if you can use that function to download the files from a git repository or it needs to be an html server.
If you can show raw content of file, you can basically download it over http(s). For example, in github, if you click “Get raw”, you her redirected to url, where you can view raw file, so you can just copy that link and download it from there. Just make sure, you try to download latest version, so that you do not have to change url. Or if you like versioning, you can somehow fetch your desired version and then fetch file based on tag.
User avatar
ivan
Party member
Posts: 1911
Joined: Fri Mar 07, 2008 1:39 pm
Contact:

Re: Wrapper / abstraction layer mechanism that download needed files on demand

Post by ivan »

I have to admit that fetching files from Github is a pretty cool idea.
There are security implications and HTTPS is not supported out of the box by Love2D.
Also, downloading files could be done without blocking the main thread using luasocket with settimeout(0)
So yes, it can be done but doing it the "proper" way is not simple.
Also you cannot easify replace any of your game's Lua files especially if it is packaged or fused.
Andlac028
Party member
Posts: 174
Joined: Fri Dec 14, 2018 2:27 pm
Location: Slovakia

Re: Wrapper / abstraction layer mechanism that download needed files on demand

Post by Andlac028 »

ivan wrote: Mon Jan 23, 2023 7:53 pm There are security implications and HTTPS is not supported out of the box by Love2D.
Actually, it is now not hard to use https, as it will be implemented in 12.0 via lua-https and you can also use prebuild library from here for older versions (just put it in folder with your application and require it)

Also for files, I think that you can store them in some data directory (writable by love.filesystem.write), correctly set append path in conf, so it will override files from source (not sure if applicable also for main.lua and conf.lua)
glitchapp
Party member
Posts: 235
Joined: Tue Oct 05, 2021 10:34 am
Contact:

Re: Wrapper / abstraction layer mechanism that download needed files on demand

Post by glitchapp »

Andlac028 wrote: Mon Jan 23, 2023 8:27 pm
ivan wrote: Mon Jan 23, 2023 7:53 pm There are security implications and HTTPS is not supported out of the box by Love2D.
Actually, it is now not hard to use https, as it will be implemented in 12.0 via lua-https and you can also use prebuild library from here for older versions (just put it in folder with your application and require it)

Also for files, I think that you can store them in some data directory (writable by love.filesystem.write), correctly set append path in conf, so it will override files from source (not sure if applicable also for main.lua and conf.lua)
I agree that security is important, however, first I need to make it work and then will take care of using https or make it work on a different thread.

Further information about this issue is here: https://codeberg.org/glitchapp/fish-fil ... e/issues/6

The size of the assets which reached a peak of 2,4gb has been reduced thanks to the implementation of a webp library and a good compromise in the audio quality and now is around 600mb. 600mb is still too big and a barrief for many people and this project would benefit a lot from a lazyloading mechanism since many people won't beat most of the levels or would only test the game.

The lazyloading.lua file will be added on the next update, any contributions are welcome.
Last edited by glitchapp on Wed Jan 25, 2023 7:52 am, edited 1 time in total.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 12 guests