Opened 13 years ago

Closed 13 years ago

Last modified 13 months ago

#8545 closed patch

DXA: New compression scheme for encode_dxa

Reported by: johndoe123 Owned by: Kirben
Priority: normal Component: Tools
Keywords: Cc:
Game:

Description

Included is a patch that adds a new experimental
compression scheme to the Dxa encoder.

I tested it with the Dxa videos from the German 4CD
version of Feeble Files. Compared to the current
encode_dxa the total Dxa size (without audio) was about
350 MB smaller with the new scheme.
See the other item for a player.

Now a rough description how it works:

Each frame is divided into 4x4 blocks. For each block
there's a block type and some data.
The frame data is additionally compressed with zlib.

Block types:
0: skip block
1: diff block (full)
2: solid color block
3: full block
4: motion vector block
10-15: diff block (one byte)

skip block:
simply skips the current block (i.e. it's unchanged
from the previous frame).

solid color block:
fill the current block with the next byte.

full block:
fill the current block with the next 16 bytes.

motion vector block:
read the next byte.
the byte has the bitformat "axxxbyyy", where xxx and
yyy are the relative offsets from
the current block position from where the block should
be copied from.
bits a and b are the sign bits for the x and y offsets.

diff block (full):
read the next unsigned short (little endian).
for each bit in the value (from most to least
significant bit), if the bit is set,
read one pixel, otherwise go to the next pixel.

diff block (one byte):
"one byte" is basically the same as "full" with one
difference, here
max. 8 pixels are changed (see source for how it works :)

Ticket imported from: #1504492. Ticket imported from: patches/650.

Attachments (5)

dxa.patch (8.8 KB ) - added by johndoe123 13 years ago.
Patch for encode_dxa
dxa_player.zip (6.2 KB ) - added by johndoe123 13 years ago.
Dxa player source
dxa_v2.patch (6.1 KB ) - added by Kirben 13 years ago.
Patch against current SVN
dxa-unification.diff (16.4 KB ) - added by eriktorbjorn 13 years ago.
Patch against current SVN
dxa-unification_v2.diff (22.4 KB ) - added by Kirben 13 years ago.
Updated patch

Download all attachments as: .zip

Change History (20)

by johndoe123, 13 years ago

Attachment: dxa.patch added

Patch for encode_dxa

by johndoe123, 13 years ago

Attachment: dxa_player.zip added

Dxa player source

comment:1 by johndoe123, 13 years ago

Added a standalone player that incorporates the new scheme.
It's based on the player found at
http://membres.lycos.fr/cyxdown/scummvm/feeble/

comment:2 by fingolfin, 13 years ago

Interesting... though I wonder if we shouldn't change the format after such a big
change :)

by Kirben, 13 years ago

Attachment: dxa_v2.patch added

Patch against current SVN

comment:3 by Kirben, 13 years ago

We would get a reduced file size for the cutscenes of
Broken Sword 1 and 2 too.

Current compression (Video only):
71.6MB (Broken Sword 1)
145MB (Broken Sword 2)

New compression (Video only):
65.8MB (Broken Sword 1)
109MB (Broken Sword 2)

comment:4 by sev-, 13 years ago

The I highly recommend to commit this patch to trunk and
make correspondent modifications to our player.

As I understand, codebase will grow even more, so is code
duplicate. May be it will be the good time to unify this
code a bit between HE games and Simon engine. Later when
someone will implement DXA player for sword engines, he will
use this library.

comment:5 by fingolfin, 13 years ago

I wrote: " I wonder if we shouldn't change the format after
such a big change :)"

I actually meant that I was wondering whether it is
appropriate to keep the name DXA if the format changes. In
particular, are old DXA files still compatible with the new
scheme? Not everybody will want to recompress his existing
FF files. The ideal situation would be if we could support
both "old style" and "new style" DXA files seemlessly.

Also, I wonder if anybody tested this on a slower machine
(Torbjörn)... :-).

Anyway, as Eugene said, indeed it would be nice if the code
for DXA playback could be unified into one (or several)
shared files in the graphics directory.

Finally, those size savings are indeed quite impressive,
esp. for BS2.

comment:6 by Kirben, 13 years ago

Well we are still using the DXA structure, but expanded to
cover an additional compression type (type 12).

The original compression types (type 2 - 3) are still
unchanged and available for use. The original compression
types will not be removed, considering they are used by
Amiga and Macintosh versions of the Feeble Files.

The new compression type, reduces the file size of the 2CD
version of the Feebles Files for Windows by a large amount
too.

New compression:
757MB (With FLAC audio files)

Current compression:
1.03G (With FLAC audio files)

comment:7 by eriktorbjorn, 13 years ago

The patch for dxa_encode does not apply cleanly any longer.

comment:8 by eriktorbjorn, 13 years ago

I made some educated guesses (not attached here) about the
rejected parts of the encode_dxa patch, and used the result
to compress the Broken Sword 1 cutscenes. The resulting
files contain plenty of type 12 frames, and play fine with
dxaply, even while encode_dxa running in the background.

CPU requirements for the new cutscenes seem to be about the
same as for the old. (Which wasn't that high to begin with.)
Perhaps a tad lower in at least some cases.

comment:9 by eriktorbjorn, 13 years ago

Sorry, the patch does apply cleanly. I was using the first
version. (For some reason, I assumed that the correct patch
was for playback in ScummVM, not for the tool itself.)

comment:10 by eriktorbjorn, 13 years ago

Extracting and compressing the video only (I've already
converted the audio) from the 2 CD version takes somewhere
between 35 and 40 hours on my 450 MHz P3, but the space
saving is considerable:

Original DXA encoder: 1096 MB
Current DXA encoder: 945 MB
Proposed DXA encoder: 640 MB

And while it's not an entirely fair comparision, it's still
nice to note that with Ogg Vorbis-encoded audio (at default
quality, i.e. 3), the resulting cutscenes are not much
larger than the original Smacker ones.

CPU requirements for playback still seem about the same as
with the current encoder's DXA movies.

by eriktorbjorn, 13 years ago

Attachment: dxa-unification.diff added

Patch against current SVN

comment:11 by eriktorbjorn, 13 years ago

I've attached a patch that adds the new decompression
function to ScummVM's DXA player, and goes some way towards
unifying the Simon and SCUMM players. There's still work to
be done on the latter, and I've only tested the Simon
player. (I don't have any of the affected HE games.)

comment:12 by Kirben, 13 years ago

I attached updated patch, with main code of
loadFile/closeFile functions merged too.

by Kirben, 13 years ago

Attachment: dxa-unification_v2.diff added

Updated patch

comment:13 by sev-, 13 years ago

Owner: set to Kirben
Status: newclosed

comment:14 by sev-, 13 years ago

Code was committed to SVN

comment:15 by digitall, 13 months ago

Component: Tools
Note: See TracTickets for help on using tickets.