3DS: make use of linear heap to make resource-intensive games playable

[BallM4788]

The 3DS has two types of heaps: a regular heap and a linear heap. ​Here is a visualization. Regular heap is used via the typical methods (new, malloc, calloc, realloc, free, etc.), while linear heap must be accessed via ​these ​functions provided by ctrulib.

On the O3DS (Original 3DS), the main program memory, the regular heap, and the linear heap combined can be no greater than 64MB by default; this can be increased up to 96MB when building ScummVM as a .cia file by ​editing the .rsf file. Similarly, on the N3DS (New 3DS), the combined memory cannot exceed 124MB by default, but can be increased to 178 by ​editing the .rsf file. However, per internal 3DS documentation from the Nintendo gigaleaks (find it yourself), the regular heap is capped to 96MB, even on the N3DS.

Currently, the 3DS port of ScummVM uses the default regular and linear heap sizes ​as defined in ctrulib (<=24MB and <=32MB, respectively). Not only does it mean that there is tons of available memory currently being unused (up to 152+MB, not accounting for stack memory and screen buffers in the linear heap), but it also means the TinyGL-powered games all segfault during initial loading, if not immediately. As far as I know, the only 3D game that can currently be run on the 3DS port is Blade Runner, and that's only because it uses voxel-based character models.

I have experimented with maxxing out the regular heap to its 96MB cap, and in both Grim Fandango and The Longest Journey, the intro cutscenes play in their entirety and the starting areas load up successfully (most of Manny's office in GF, and the Arcadia cliffside in TLJ). Attempting to progress beyond this (attempting to read the memo in the mail tube, accessing Manny's inventory, moving right to the screen with his computer, or leaving his office in GF; walking right to the White Dragon's nest or further in TLJ) causes a segfault.

If the 3DS's linear heap could be used to hold memory-intensive data, such as textures, streaming data, etc., it might very well be possible to get all the 3D games to run in the 3DS port. Except for Penumbra Overture, of course. That would be a fool's errand.

[ccawley2011]

Most memory allocation in TinyGL is done using the wrapper functions in graphics/tinygl/memory.cpp, so a good way to get started is to edit that file to use the linear allocation functions instead of the stdlib ones and see how well it works.

[BallM4788]

Altered graphics/tinygl/memory.cpp accordingly, but no change in overall behavior. I'm linking relevant files that might help decide where to go from here:

​Diff of my changes
​Build warnings + build flags (no different than usual)
​GDB log of Grim crash
​GDB log of Stark crash

[BallM4788]

Replying to ccawley2011:

Most memory allocation in TinyGL is done using the wrapper functions in graphics/tinygl/memory.cpp, so a good way to get started is to edit that file to use the linear allocation functions instead of the stdlib ones and see how well it works.

See my comment above.
Also, I tested out ​your pull request in combination with my changes above, and the results were as follows:

• For Grim Fandango, the screen remains black, and doesn't seem to ever finish loading. No crashing though, it just stays stuck on the black screen.
• For The Longest Journey, the screen also remains black, but stuff is loading behind the scenes as all the audio plays properly.
• After launching either game, if the user attempts to bring up the pause menu, the screen remains blank.

I have no clue why putting TinyGL's Z-buffer on the linear heap would cause the screen to go blank. It clearly has no problem putting texel buffers on the linear heap...

[neuromancer]

Can you test with Driller as well? It should be the less demanding 3D game in scummvm