Opened 13 years ago

Closed 13 years ago

Last modified 17 months ago

#8594 closed patch

DXA: Improved encoder (filesize and speed)

Reported by: johndoe123 Owned by: Kirben
Priority: normal Component: Tools
Keywords: Cc:
Game:

Description

Hi,

Long promised to Kirben I finally finished the new and improved version of the DXA encoder.

The changes include: - Frame type 12 was removed and replaced by type 13 - The new type produces files a lot smaller (depending on the input, of course) - Encoding should be faster (if only a bit) - The height scaling from Smacker animations (interlacing and line doubling) is now taken into account, too

The DXA files of the current SVN encoder can still be used in ScummVM as long as the code for type 12 isn't removed for some reason.

The DXA files from Feeble Files are now about 1.0 GB big, the Smacker animations 1.4 GB. I also compressed the videos from Broken Sword 2, the Smacker files are 130 MB big, the DXA files 96 MB. (Note that these do not include the audio files.)

Attached is the patch for the current SVN version, a win32 exe of the encoder and source/exe for the standalone DXA player (needs SDL and doesn't play files produced by the "old" encoder).

I haven't added the new code to the ScummVM DXA player yet as I wanted to hear what you think first.

Now for some technical details: The new type 13 uses the basic ideas from type 12 but adds two new block types:

1. Blocks with 4 colors or less now use a bitmap for each pixel and store the pixel data. E.g. if a block has only two colors, it stores a 16-bit bitmap and two bytes for each actual pixel. Now bit 0 in the map means 'use pixel 1' while bit 1 means 'use pixel 2'.

2. Blocks are now also divided into 4 subblocks, each 2x2 pixels. For each of the subblocks the same methods as for a 4x4 block are tried and the smallest one is chosen.

Also the data layout has been modified to improve compression with zlib. Before the data has been stored sequentially (e.g. "block type, block data, block type" etc.) which hurt compression. Now each type of data is stored in its own "section" which improves compression a lot.

Encoding should also be faster now since only so much data is compressed until compression wouldn't gain anything. This is simply done by limiting the available output size to zlib's compress2 function to the size of the previously compressed data (in the order type 13->xor->full).

The CPU requirements shouldn't be drastically different.

Ticket imported from: #1611503. Ticket imported from: patches/699.

Attachments (5)

encode_dxa.patch (21.7 KB ) - added by johndoe123 13 years ago.
The patch file
dxa_player.zip (43.8 KB ) - added by johndoe123 13 years ago.
Player source and exe
encode_dxa.zip (97.8 KB ) - added by johndoe123 13 years ago.
Encoder exe
dxa_update.diff (8.2 KB ) - added by Kirben 13 years ago.
Patch for ScummVM SVN
dxa_endian_safe_diff.zip (1.6 KB ) - added by cyxx 13 years ago.
(untested) endianness patch

Download all attachments as: .zip

Change History (15)

by johndoe123, 13 years ago

Attachment: encode_dxa.patch added

The patch file

by johndoe123, 13 years ago

Attachment: dxa_player.zip added

Player source and exe

comment:1 by johndoe123, 13 years ago

File Added: dxa_player.zip

by johndoe123, 13 years ago

Attachment: encode_dxa.zip added

Encoder exe

comment:2 by johndoe123, 13 years ago

File Added: encode_dxa.zip

comment:3 by fingolfin, 13 years ago

Sounds very nice, I must say. Unfortunately I can't work on this patch ("work" as in 'review, accept or reject it'), as I have no good way to test it. But definitely sounds good.

Hm, I just noticed we list you twice in our contributors list: Once with your real name, once under your nick... Guess I'll fix that now ;-)

comment:4 by Kirben, 13 years ago

File Added: dxa_update.diff

by Kirben, 13 years ago

Attachment: dxa_update.diff added

Patch for ScummVM SVN

comment:5 by Kirben, 13 years ago

File Added: dxa_update.diff

comment:6 by Kirben, 13 years ago

Owner: set to Kirben
Status: newclosed

comment:7 by Kirben, 13 years ago

Excellent work, the DXA compression improvements offer much better fize sizes, compared to previous DXA compression used too.

Broken Sword 1: Old: 65.8MB New:45.4MB

Broken Sword 2: Old: 109MB New: 95.9MB

Feeble Files 4CD: Old: 1.25GB New: 1.05GB

I added the DXA encoder and decoder improvements to ScummVM SVN, keeping type 12 for backwards compatibility.

by cyxx, 13 years ago

Attachment: dxa_endian_safe_diff.zip added

(untested) endianness patch

comment:8 by cyxx, 13 years ago

It looks like the encoder and player code aren't endian safe. I don't know if that's a deliberate choice, but if it isn't, the attached patches ais at "fixing" this. I choose to encode everything as big-endian to match the endianness of the original DXA tags.

I haven't "fixed" block type 12, since the code for encoding it isn't present anymore in encode_dxa.

File Added: dxa_endian_safe_diff.zip

comment:9 by Kirben, 13 years ago

Feeble Files 2CD: Old: 639MB New: 550MB

I added patches for endian issues to ScummVM SVN, encoding as big endian as suggested.

comment:10 by digitall, 17 months ago

Component: Tools
Note: See TracTickets for help on using tickets.