Opened 14 years ago

Closed 14 years ago

Last modified 2 years ago

#8545 closed patch

DXA: New compression scheme for encode_dxa

Reported by: johndoe123 Owned by: Kirben
Priority: normal Component: Tools
Keywords: Cc:
Game:

Description

Included is a patch that adds a new experimental compression scheme to the Dxa encoder.

I tested it with the Dxa videos from the German 4CD version of Feeble Files. Compared to the current encode_dxa the total Dxa size (without audio) was about 350 MB smaller with the new scheme. See the other item for a player.

Now a rough description how it works:

Each frame is divided into 4x4 blocks. For each block there's a block type and some data. The frame data is additionally compressed with zlib.

Block types: 0: skip block 1: diff block (full) 2: solid color block 3: full block 4: motion vector block 10-15: diff block (one byte)

skip block: simply skips the current block (i.e. it's unchanged from the previous frame).

solid color block: fill the current block with the next byte.

full block: fill the current block with the next 16 bytes.

motion vector block: read the next byte. the byte has the bitformat "axxxbyyy", where xxx and yyy are the relative offsets from the current block position from where the block should be copied from. bits a and b are the sign bits for the x and y offsets.

diff block (full): read the next unsigned short (little endian). for each bit in the value (from most to least significant bit), if the bit is set, read one pixel, otherwise go to the next pixel.

diff block (one byte): "one byte" is basically the same as "full" with one difference, here max. 8 pixels are changed (see source for how it works :)

Ticket imported from: #1504492. Ticket imported from: patches/650.

Attachments (5)

dxa.patch (8.8 KB ) - added by johndoe123 14 years ago.
Patch for encode_dxa
dxa_player.zip (6.2 KB ) - added by johndoe123 14 years ago.
Dxa player source
dxa_v2.patch (6.1 KB ) - added by Kirben 14 years ago.
Patch against current SVN
dxa-unification.diff (16.4 KB ) - added by eriktorbjorn 14 years ago.
Patch against current SVN
dxa-unification_v2.diff (22.4 KB ) - added by Kirben 14 years ago.
Updated patch

Download all attachments as: .zip

Change History (20)

by johndoe123, 14 years ago

Attachment: dxa.patch added

Patch for encode_dxa

by johndoe123, 14 years ago

Attachment: dxa_player.zip added

Dxa player source

comment:1 by johndoe123, 14 years ago

Added a standalone player that incorporates the new scheme. It's based on the player found at http://membres.lycos.fr/cyxdown/scummvm/feeble/

comment:2 by fingolfin, 14 years ago

Interesting... though I wonder if we shouldn't change the format after such a big change :)

by Kirben, 14 years ago

Attachment: dxa_v2.patch added

Patch against current SVN

comment:3 by Kirben, 14 years ago

We would get a reduced file size for the cutscenes of Broken Sword 1 and 2 too.

Current compression (Video only): 71.6MB (Broken Sword 1) 145MB (Broken Sword 2)

New compression (Video only): 65.8MB (Broken Sword 1) 109MB (Broken Sword 2)

comment:4 by sev-, 14 years ago

The I highly recommend to commit this patch to trunk and make correspondent modifications to our player.

As I understand, codebase will grow even more, so is code duplicate. May be it will be the good time to unify this code a bit between HE games and Simon engine. Later when someone will implement DXA player for sword engines, he will use this library.

comment:5 by fingolfin, 14 years ago

I wrote: " I wonder if we shouldn't change the format after such a big change :)"

I actually meant that I was wondering whether it is appropriate to keep the name DXA if the format changes. In particular, are old DXA files still compatible with the new scheme? Not everybody will want to recompress his existing FF files. The ideal situation would be if we could support both "old style" and "new style" DXA files seemlessly.

Also, I wonder if anybody tested this on a slower machine (Torbjörn)... :-).

Anyway, as Eugene said, indeed it would be nice if the code for DXA playback could be unified into one (or several) shared files in the graphics directory.

Finally, those size savings are indeed quite impressive, esp. for BS2.

comment:6 by Kirben, 14 years ago

Well we are still using the DXA structure, but expanded to cover an additional compression type (type 12).

The original compression types (type 2 - 3) are still unchanged and available for use. The original compression types will not be removed, considering they are used by Amiga and Macintosh versions of the Feeble Files.

The new compression type, reduces the file size of the 2CD version of the Feebles Files for Windows by a large amount too.

New compression: 757MB (With FLAC audio files)

Current compression: 1.03G (With FLAC audio files)

comment:7 by eriktorbjorn, 14 years ago

The patch for dxa_encode does not apply cleanly any longer.

comment:8 by eriktorbjorn, 14 years ago

I made some educated guesses (not attached here) about the rejected parts of the encode_dxa patch, and used the result to compress the Broken Sword 1 cutscenes. The resulting files contain plenty of type 12 frames, and play fine with dxaply, even while encode_dxa running in the background.

CPU requirements for the new cutscenes seem to be about the same as for the old. (Which wasn't that high to begin with.) Perhaps a tad lower in at least some cases.

comment:9 by eriktorbjorn, 14 years ago

Sorry, the patch does apply cleanly. I was using the first version. (For some reason, I assumed that the correct patch was for playback in ScummVM, not for the tool itself.)

comment:10 by eriktorbjorn, 14 years ago

Extracting and compressing the video only (I've already converted the audio) from the 2 CD version takes somewhere between 35 and 40 hours on my 450 MHz P3, but the space saving is considerable:

Original DXA encoder: 1096 MB Current DXA encoder: 945 MB Proposed DXA encoder: 640 MB

And while it's not an entirely fair comparision, it's still nice to note that with Ogg Vorbis-encoded audio (at default quality, i.e. 3), the resulting cutscenes are not much larger than the original Smacker ones.

CPU requirements for playback still seem about the same as with the current encoder's DXA movies.

by eriktorbjorn, 14 years ago

Attachment: dxa-unification.diff added

Patch against current SVN

comment:11 by eriktorbjorn, 14 years ago

I've attached a patch that adds the new decompression function to ScummVM's DXA player, and goes some way towards unifying the Simon and SCUMM players. There's still work to be done on the latter, and I've only tested the Simon player. (I don't have any of the affected HE games.)

comment:12 by Kirben, 14 years ago

I attached updated patch, with main code of loadFile/closeFile functions merged too.

by Kirben, 14 years ago

Attachment: dxa-unification_v2.diff added

Updated patch

comment:13 by sev-, 14 years ago

Owner: set to Kirben
Status: newclosed

comment:14 by sev-, 14 years ago

Code was committed to SVN

comment:15 by digitall, 2 years ago

Component: Tools
Note: See TracTickets for help on using tickets.