SCI: Some Macintosh games are very slow to start

[eriktorbjorn]

I've noticed that the Macintosh versions of Leisure Suit Larry 6 and 7 take a long time to start in ScummVM. Particularly the first time. Subsequent startups are faster, presumably because the operating system has cached the files it's reading.

But the first time it was slow enough that I thought for sure that ScummVM had locked up somehow.

The problem seems to be ResourceManager::readResourcePatchesBase36(), where it loops through a bunch of files, patching them somehow. For Leisuire Sute Larry 6, it goes through 4,917 files. For Leisure Suit Larry 7, it's 3,807. Would it be possible to do this processing on demand, rather than all in advance, or something like that?

I have Macintosh versions of Phantasmagoria 1 and Gabriel Knight 2, but I don't know how to install those for ScummVM so I haven't tested if they're similarly slow to start.

Also, is there any purpose to this part of the function?

				// Check for SOL as well
				tag = (tag << 16) | stream->readUint16BE();

				if (tag != MKTAG('S','O','L',0)) {
					delete stream;
					continue;
				}

				delete stream;

[m-kiewitz]

The problem is that we need to know which resource to use. If we have to use a patch resource or not.
Are there really that many patch files for these games?
Maybe there is a subdirectory which has files that are detected as potential patch files, but they are not patch files?

Another problem is if we do it on demand, we would have to search through quite a few directories PER FILE, and that would be a performance problem while the game is running.

[m-kiewitz]

Oh and the problem is that originally you couldn't accidentally create a filename for a resource patch, but with SCI32 that changed and now basically almost any filename can be considered a script patch, which is why we have to check potentially thousands of files if they are or if they are not.

[m-kiewitz]

And I mean you have to understand:
For *ANY* resource access, we would have to check for a resource patch in your case.
This means tons of stutters everywhere.

Even if we read out the directories and check the individual files only on access, that could still introduce pseudo-random stutters in games, which would get reported as bugs.

[m-kiewitz]

Oh, also it's not even just resource access.
I think there are even situations where a script for example checks if a resource exists.
And you can in fact add additional resources via patch files. It's not even just existing resources, it may be new resources.

Even just remembering each resource that you already checked would also increase memory requirements.

You really have to check what's going on, either there are thousands of patch files, or it's just tons of files that have nothing to do with the game and are just checked.

[eriktorbjorn]

Thanks for the explanations. Here are the file lists from my Macintosh discs:

Leisure Suit Larry 6: ​https://drive.google.com/drive/u/0/folders/1Jw72TIO8syKEHP6BZmyvDLBhYaBlOOoz

Leisure Suit Larry 7: ​https://drive.google.com/drive/u/0/folders/1dgE0HVz1jY3S11NaXcldzBmrjY03m3Xo

Phantasmagoria 1: ​https://drive.google.com/drive/u/0/folders/1akJ4ZIaqJ3WYz8zmydJhTlb4ZP3JNnfn

Gabriel Knight 2: ​https://drive.google.com/drive/u/0/folders/1OdUdNiScguXchr6WQc55DPKMgw-rDYib

Please let me know if there are any files you need to know more about.

The startup times for Leisure Suit Larry 6 and 7 are 25-30 seconds on my computer the first time. If I try it again, while everything is presumably still in the operating system's disc cache, it's almost instantaneous. Maybe it would be faster on an SSD disc (not to mention a newer computer), but I only have a relatively small one. The bulk of my data, including my ScummVM games, is on a traditional hard disc.

[m-kiewitz]

Yikes.
It seems it's all standalone audio files for some reason.
I guess on Mac there's no resource.aud-like file?

Can you please check if the audio files contain the SOL header?
You can easily check the Larry 6 files in AIFF folder.

But I have to wonder why the access is so slow. Why does it take long to read the first few bytes of 4000 files?

[eriktorbjorn]

The "file" command identifies all the LSL6 files in the Voices directory (was that what you meant?) as "IFF data, AIFF audio". Looking at one at random, this is how it starts:

00000000: 464f 524d 0001 6fc2 4149 4646 434f 4d4d  FORM..o.AIFFCOMM
00000010: 0000 0012 0001 0001 6f94 0008 400d ac44  ........o...@..D
00000020: 0000 0000 0000 5353 4e44 0001 6f9c 0000  ......SSND..o...
00000030: 0000 0000 0000 0505 0506 0505 0707 0709  ................

For the LSL7 Voices Folder directory, they do have a SOL header it seems. Here's the start of one I picked at random:

00000000: 0d0c 534f 4c00 2256 0d65 8d01 0000 0000  ..SOL."V.e......
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................

[m-kiewitz]

Normally there is a resource.aud with all the audio resources inside of it.
That also means fast + easy access and one directory that can get read fast.

On Mac for these SCI32 games it seems Sierra decided to not do that, and instead have all audio resources in its own file.

A proper solution to this, well we would have to check why the performance is so bad first, but if we can't do anything about that, we could write a tool which would put all these audio resources inside a big file.

[m-kiewitz]

You are trying to play this on a Mac?
Or on PC?
If on PC, does the PC have a SSD or a regular drive?
I'm really suprised why a somewhat modern PC would have such a slow file access.

Can you specify how long it takes to start up?

[eriktorbjorn]

Linux. As I said, this is a regular drive. I don't have space on my SSD drive for all my ScummVM games. (Well, not without some serious re-arranging, at least.) This computer was bought at a time when it made sense to have a relatively small SSD drive that the computer would boot from and for daily work, and a larger regular one for storage.

I don't remember which year I got the computer. It must be getting close to 10 years ago, but not quite... 8, perhaps? 9? I've been meaning to get a new one, but then there were some unforseen (and some foreseen) expenses that pushed that into the future.

[eriktorbjorn]

I tried copying just these two games to the SSD disk, and did some magic commands to hopefully clear the disc cache.

From SSD, the startup time appears almost instantaneous. But I should try restarting the computer to see if I really did clear the disc cache.

[m-kiewitz]

Lucky me, I bought Space Quest 6 Mac German a few weeks ago.
That game should maybe have the same issues like yours.
I haven't had time to look into it. Not even sure if I can read a Mac-CD using a regular Windows PC.

There may be performance issues even inside ScummVM. I remember there being a really weird performance issue with some SCI games, and in the end it was the dir() command, and LordHoto added a cache to it, which in turn caused other issues, like you not being able to modify directories anymore without restarting ScummVM. But that solved the performance.

I think it was some SCI game that queried a file 1000 times a second or some other nonsense like that.
Still, even in that case I would have expected the OS to cache the directory listing. It's on a local drive.

[eriktorbjorn]

Re-confirmed after reboot:

Startup time from SSD is almost instantaneous for either game.
Startup time from HDD is about half a minute.

So I guess the problem isn't quite as bad as I thought, but if there's something simple that can be done to optimize it that wouldn't hurt...

[eriktorbjorn]

I haven't had time to look into it. Not even sure if I can read a Mac-CD using a regular Windows PC.

I assume there are tools that can make ISO images out of them, at least. After that, ​https://www.scummvm.org/dumper-companion should do the trick.

[m-kiewitz]

That tool was really nice, I got Space Quest 6 for Mac now.
It's even the English version, just the CD cover says SQ6 German, but it also says DOS/Windows on the back.

Anyway, they did the same for this one.
There are even 9 different aud-folders.

Why couldn't they just put it all inside one big file?

[m-kiewitz]

Space Quest 6 English Mac screenshot

[eriktorbjorn]

Why couldn't they just put it all inside one big file?

I can only assume that felt this way was either more efficient or easier on the original hardware.

[m-kiewitz]

What could be done for this situation is to not look through all audio files separately, but go basically: "this is a mac SCI32 version, so for these directories, just look through the folders and assume that all of these files are in fact audio resources"

I guess the Sierra interpreter just handled these sub directories like a resource.aud file and not as regular patch files.

[sluicebox]

Duplicate of #11353

[m-kiewitz]

Replying to eriktorbjorn:

Why couldn't they just put it all inside one big file?

I can only assume that felt this way was either more efficient or easier on the original hardware.

But they bundled the other resources inside those data* containers.
why not make audio* containers too?
Or maybe those data* containers are Mac standard, I have no idea. Okay, if that's the case, put the audio in there too.
It really makes no sense to me.

I even wonder if Macs had stutters because of these thousands of files.
I remember Mortal Kombat 2 on PC doing something similar and the game didn't run well at all either.

Almost feels as if some different internal group or even company ported the Mac games.

[m-kiewitz]

I think clone2727, a great ScummVM developer who sadly left the team, added tons of Mac support ages ago, including adding support for these special menu bars.

Adding the audio files as patches was probably the most easiest route with the least amount of changes to the source code and it works. But as I said, maybe there should be additional code handling these special directories for Mac versions.

For my Space Quest 6 Mac it's "Voices Folder" and sub folders "Aud1" up to "Aud9".
It seems the folders are at least the same, but the contents aren't.

For Space Quest 6 it's also AIFF files without SOL header. So weird.

Offset 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
00000000 46 4F 52 4D 00 01 7A 1C 41 49 46 46 43 4F 4D 4D FORM..z.AIFFCOMM
00000016 00 00 00 12 00 01 00 01 79 EE 00 08 40 0D AC 44 ........yî..@.¬D
00000032 00 00 00 00 00 00 53 53 4E 44 00 01 79 F6 00 00 ......SSND..yö..

[eriktorbjorn]

I did only a tiny bit of programming (in Pascal) on a Mac, ages ago. My impression from looking at other programs was that it was a bit of a mix.

Sometimes there were straight data files, sometimes the data was put in the resource fork (seems like a natural way to implement an archive to me), sometimes the resource fork was the excutable code and the data fork was the game data... whichever was the most convenient, I guess.

[eriktorbjorn]

I forgot to mention that I also have the Mac version of Castle of Dr. Brain. It does not have this excessive number of files, so I guess it's just the later talkie games.

[m-kiewitz]

yes, Castle of Dr. Brain has no resource.aud