Opened 17 years ago

Closed 16 years ago

Last modified 19 months ago

#8162 closed patch (outdated)

COMI lip sync patch (experimental)

Reported by: SF/dschepler Owned by: aquadran
Priority: normal Component: Engine: SCUMM
Keywords: Cc:
Game: Monkey Island 3


This is a first version of a patch to implement lip syncing for COMI. I don't know whether the lip positions are exactly correct, but they do seem to move roughly in sync with the speech.

Ticket imported from: #671104. Ticket imported from: patches/267.

Attachments (1)

lipsync.diff (4.7 KB ) - added by SF/dschepler 17 years ago.
Lip sync patch

Download all attachments as: .zip

Change History (10)

by SF/dschepler, 17 years ago

Attachment: lipsync.diff added

Lip sync patch

comment:1 by fingolfin, 17 years ago

First off, thanks for this :-)

Now, an idea on how to improve this possibly:

in voiceLipSyncData() you do a binary ssearch time it is called to find the correct lip sync entry. However, why not make use of the fact that the sound playback is always forward: initially, lipsync entry 0 is going to be used, then after the first threshold, entry 1, then entry 2 etc... so if you buffer the address of the last useed entry, and then do a linear search starting from that, it should actually perform better than doing a binary search each time (that is, it will usually find the correct entry in near constant time: it's either the current or the next entry).

I need to actually test this and look at the code more before I can make any other comments :-)

comment:2 by SF/dschepler, 17 years ago

Actually, when I put in debugging statements, it turned out the lip sync functions were only getting called every 4 or 5 jiffies. Still, you're right on the linear search. I'll make that change when I get some time.

I also had reports on #scummvm that to other people the lip positions didn't look right -- Guybrush made the O position too often. I don't really know how to fix this -- switching width and height made it look to me like he was mumbling. There is a big-endian uint32 at the beginning of each SYNC data block which I'm just skipping for now; maybe it needs to be figured in somehow, but I'd have no idea how.

comment:3 by SF/dschepler, 17 years ago

Also, it just occurred to me that I should probably somehow have the timing tied to where the playback actually is instead of wall clock time, in case the actor frequency is set.

comment:4 by SF/ender, 17 years ago

Yeah, the results look pretty schewed here. Let me know when you have an improved version, and we'll probably stick that in CVS :)

comment:5 by fingolfin, 17 years ago

Would be nice if this work doesn't go to the waste bin...

comment:6 by fingolfin, 16 years ago

Some notes:

* SYNC may also occur in music data. No surprise, since there are also opcodes for musicLipSyncWidth/Height * the first 4 bytes in the SYNC are equal to the first 4 bytes in the REGN block, and is simply the offset in bytes to the start of the DATA block, I think. So ignoring them should be safe

* In the patch, a jiffie is assumed to be 17ms. This isn't explained anywhere, but comparing with the rest of ScummVM; I'd guess 1000/60 = 16.666... is where this is from. In any case, I made some experiements (computed the ratio between data size, highest lip sync 'offset', and the sample size), which seem to confirm a jiffie length of 16-17ms

* Yes, the byte offset of the sound should be computed by looking at how many bytes are already used up, exactly. This is not easily possible when playRaw is used. However, using a custom AudioInputStream, this should be 'easily' possible to fix.

comment:7 by fingolfin, 16 years ago

Owner: set to aquadran
Resolution: outdated
Status: newclosed

comment:8 by fingolfin, 16 years ago

Pawel implemented proper lip sync support for both speech and music (I believe) in the new rewritten iMuse Digital. Enjoy :-)

comment:9 by digitall, 19 months ago

Component: Engine: SCUMM
Game: Monkey Island 3
Note: See TracTickets for help on using tickets.