Cloud: support for generic WebDAV storage

[picatsv]

Currently cloud functionality is based on just a few third party services, it would be nice to be able to use a custom (self-hosted, even!) cloud service to do that instead. WebDAV might be a versatile enough option to keep the feature supported for a good amount of services and for a long while, as well.

[i30817]

There is a (slower) workaround for this on android (since android disables mount and fuses, the only other way is the app internally having a webdav client which not many do).

Since the android port can use SAF paths, and since ​https://github.com/chenxiaolong/RSAF exists, you can install that and configure it (it uses rclone internally) to have some webdav remote that encompasses all the games you want to add.

Then instead of scanning in the gui - far far too slow because it's recursive and full of 'unknown Ags or director games'), you can instead login\ssh into the machine that holds the server, install scummvm there and add the games there from the command line since it's much faster, and by a very lucky accident instead of adding 'unknown Ags game' it just adds what game it thinks it is based on the filename (even if the checksums don't match).

Since I separate Ags and text games from other scummvm games I do this with (Linux):

(you need to do this in the base of your scummvm remote if you want to minimize the search and replace later)

rm -f add.log; find ScummVM ScummVM_Text/ ScummVM_AGS/ -mindepth 1 -maxdepth 1 -type d -exec echo "{}" >> add.log \; -exec scummvm -a -p"{}" >> add.log \;

you can search add.log then for games with weren't added (search for 'Added 0'), there is always some, either because you don't have a recent enough scummvm or because the game files aren't in the top level dir you're searching.

Anyway after that you have a scummvm.ini file with all games in ~/.config/scummvm/scummvm.ini

copy that to your android device, and in there go to the android scummvm global options, then try to open a file and go up in the filechooser until you see the line "add a remote path" choose that, then if you have rsaf active (you probably want to give it permission to run in the background and without memory limits\without memory optimization in android, it's a option inside the rsaf app), there will be a option to connect to the rsaf remote either in front of you or if you click on the 3 lines in the top left.

Do so and "authorize" that base dir.

The scummvm file chooser dialog will appear with a strange path (for me it's /saf/games%3A because my remote is called games). Note it down. Then go into global options, backend and export configuration (this saves a zip file in your android device somewhere).

Then open the scummvm.ini file you transfered to your android device and find replace all

path=

by

path=/saf/games%3A

save. this turns all your paths there to paths to your games in the remote (note that this is why a told you to call find in the top level webdav dir, if the paths there are absolute or something like that, you need to replace a larger party than just "path=" until the paths match the SAF webdav paths).

Find the configuration zip file you saved and replace the scummvm.ini file you saved with the newly edited ini file. Then import that configuration.

If everything went right, you'll have your whole collection of games pointing to the webdav remote. If something went wrong it's typically you trying this without adding the remote path to android scummvm with that add game just to authorize the remote webdav path thing, typically because you reinstalled scummvm or cleared its storage.

Very annoying yeah? But much much MUCH faster than waiting for the gui to recourse into tens of thousands of directories through webdav, running in RSAF (even slower), which is the alternative.

My speed for games in this is extremely bad for larger than 1 gb games or games with lots of small files, or worse, both. But I have a very bad server computer and only a 1mb speed intranet (if I'm lucky), your situation may vary.

It's another thing altogether to get this working both inside the intranet and outside. I do have a hack that works for me, but it's so config dependent and fragile that I'm not going to say it here.

[i30817]

Btw rclone is working on accessing compressed files (like squash and zip files), so it's possible in the next rclone version (and the next RSAF after that) that you'll be able to hold a compressed squash lz4 or zstd file instead of using webdav soon, which should allow to fit more games in space limited android devices without webdav and be faster.

My Ags games collection for instance (fairly comprehensive) turns into a 60gb zstd squash file.

[i30817]

Opps. If you want this just for cloud saves, well you'll probably have to use that "workaround for it to work in the intranet and internet" because that scummvm setting probably requires a url.

What worked for me, very simplified is to use duckdns, install dnsmasq in the webdav server (and configure the config file to replace the duckdns webdav url by the IP of the webdav server, and fallback to the router DNS otherwise) and configure the android home wlan connection to use the webdav server as the "DNS server(s)" (there are two options).

This is not very nice because if android devices can't find the DNS server (for instance because the server is off) you get into a loop of attempting to connect, reconnect, disconnect, etc. You can avoid this by not replacing both DNS server IPS, so when the server is off, you fallback to the other. The problem with this is that some android devices call the two DNS servers aleatory, so sometimes you'll want to refer to the server hostname (say in termux in a ssh) and it will tell you it can't reach it (because it's using the other DNS).

[i30817]

Also I had to mess with the router to passthrough the router firewall all the server services, including webdav. I recommend configuring webdav with a long password if you go this way. RSAF allows you to set the password so you won't need it for all end programs.

[picatsv]

whoops, i thought i would've got update notifications on the issue but my inbox disagrees, didn't want to leave this just hanging there
ideally this feature would come more in handy for environments where the user has less freedom of putting filesystem duct tape together like console ports (in my hypothetical example i'd be using the psvita port but the same applies for other consoles) where i wouldn't be able to do trickery like that

[i30817]

I've seen android apps use webdav client directly before (AlreaderX is one) and it doesn't apppear like they're using SAF to do it, so it may be possible to be faster using that than using a proxy like RSAF and it would work for all platforms, as long as scummvm has a appropriate VFS implementation they can slot a webdav library and config tab (for password and address).

Unfortunately you'll always need linux\proxy tricks in the server if you want to use the same address for both the intranet and internet (which of course, you might not want to, or might want to skip altogether for the internet). Modems simply aren't made to treat addresses different depending on where they come from, and using a DNS name instead of the hostname in the intranet just makes them confused and stupid and just ignore the return from the IP pointing to themselves.

[i30817]

That doesn't mean you aren't right, I'm not trying to get a last word and doing it wrong, just showing a problem I had on a mobile platform trying the webdav thing.