Today, Boss Bob shared a very interesting tool - Rhubarb Lip Sync. This is a command-line tool that can automatically generate lip-sync animations for 2D characters based on voice recordings. It can be used in any project that requires creating lip-sync animations from existing recordings, such as computer games and animated films.
Project address: https://github.com/DanielSWolf/rhubarb-lip-sync
Check out the effect
Types of Mouth Shapes
Rhubarb Lip Sync can use six to nine different mouth shapes. The first six mouth shapes (Ⓐ-Ⓕ) are basic shapes, and they are the absolute minimum number you must draw for your character. These six mouth shapes were invented by the Hanna-Barbera studio for shows like Scooby-Doo and The Flintstones. Since then, they have evolved into the de facto standard for 2D animation and are widely used by studios such as Disney and Warner Bros.
In addition to the six basic mouth shapes, there are three extended mouth shapes: Ⓖ, ⓗ, and Ⓧ. These are optional. You can choose to draw all three, or just one or two, or not use them at all.
Ⓐ - A closed mouth shape, used for sounds pronounced as "P", "B", and "M". This is almost identical to the Ⓧ shape but with a slight pressure between the lips.
Ⓑ - A slightly open mouth with clenched teeth. This mouth shape is typically used for most consonants (such as "K", "S", "T", etc.). It is also used for some vowels, such as the "EE" sound in "bee".
Ⓒ - An open mouth shape. This mouth shape is used for vowels like "EH" (men) and "AE" (bat). Depending on the context, it is also used for some consonants. In the animation process from Ⓐ or Ⓑ to Ⓓ, this shape is also used as an intermediate transition shape. Therefore, make sure the animations ⒶⒸⒹ and ⒷⒸⒹ look smooth!
Ⓓ - A wide-open mouth shape. This mouth shape is used for vowels like "AA" (father).
Ⓔ - A slightly rounded mouth shape. This mouth shape is used for vowels like "AO" (off) and "ER" (bird). In the animation process from Ⓒ or Ⓓ to Ⓕ, this shape is also used as an intermediate transition shape. Make sure the mouth does not open wider than Ⓒ. The transitions ⒸⒺⒻ and ⒹⒺⒻ should be smooth.
Ⓕ - A pouting mouth shape. This mouth shape is used for sounds like "UW" (you), "OW" (show), and "W" (way).
Ⓖ - A mouth shape where the upper teeth touch the lower lip, used for sounds like "F" (for) and "V" (very). This extended mouth shape is optional. If your art style is detailed enough, it will greatly improve the overall appearance of the animation. If you decide not to use it, you can specify this using the extendedShapes option.
Ⓗ - This shape is used for the long "L" sound, with the tongue raised behind the upper teeth. The mouth should be at least as open as the Ⓒ shape, but not as open as the Ⓓ shape. This extended mouth shape is optional. Depending on your art style and the angle of the head, the tongue may not be visible at all. In this case, there is no need to draw this additional shape. If you decide not to use it, you can specify this using the extendedShapes option.
Ⓧ - Idle state. This mouth shape is used during pauses in speech. When your character is walking but not speaking, the same mouth shape should be used. It is almost identical to Ⓐ, but with slightly less pressure between the lips: for Ⓧ, the lips should be closed but relaxed. This extended mouth shape is optional. Whether there is any visible difference between the resting state Ⓧ and the closed speaking mouth shape Ⓐ depends on your art style and personal taste. If you decide not to use it, you can specify this using the extendedShapes option.
Usage Environment
Rhubarb Lip Sync integrates with the following applications:
Adobe After Effects
Moho and OpenToonz
Spine (provided by Esoteric Software)
Vegas Pro (provided by Magix)
Visionaire Studio
Additionally, you can use Rhubarb Lip Sync's command-line interface (CLI) to generate files in various output formats (TSV/XML/JSON).
<?xml version="1.0" encoding="utf-8"?>
<rhubarbResult>
<metadata>
<soundFile>C:\Users\Daniel\Desktop\av\hi\hi.wav</soundFile>
<duration>0.47</duration>
</metadata>
<mouthCues>
<mouthCue start="0.00" end="0.05">X</mouthCue>
<mouthCue start="0.05" end="0.27">D</mouthCue>
<mouthCue start="0.27" end="0.31">C</mouthCue>
<mouthCue start="0.31" end="0.43">B</mouthCue>
<mouthCue start="0.43" end="0.47">X</mouthCue>
</mouthCues>
</rhubarbResult>