Forced Aligment is a kind of Automatic Speech Recognition but it differs in one major respect. Rather than being given a set of possible words to recognise, the recognition engine is given an exact transcription of what is being spoken in the speech data. The system then aligns the transcribed data with the audio file, identifying which time segments in the speech data correspond best to particular words in the transcription data.
Below, an explanation of what it is and how it works. For a description of "how to align your text with audio with the help of software (i.e. WebMAUS)", see the end of this page.
Difference between spoken and written text
Ideally spoken, the written text to align with the audio is an exact copy of what was said (see image above). But often, transcriptions are a corrected, interpreted version of what was said.
If someone says "If I recognize well, I was, eh I mean, we went to the store, eh to the grocery store", it will be often transcribed as "If I recognize well, we went to the grocery store". All hesitations, internal corrections of the speaker and repetitions are skipped. The result is a grammatically correct version of what was meant, but it does differ from what was said. Forced Alignment (FA) can handle this kind of deviations between written and spoken content as long as the difference is not too big, and the order of the words in the transcript is the same as the order in which they appear in the audio recording.
Unknown words and Phonetic Transcriptions
A problem may arise if the text (and audio) contain foreign words, abbreviations and (large) numbers. In order to align, the written text is transformed into phonemes according to the language setting. But how to deal with words that are strange for the target language? Suppose you have a written phrase like "Mrs Uytendenboogaard and Mr. Chukwuemeka have a boat trip on the Bhagirathi River". This phrase combines a Dutch and a Nigerian family name with the name of an Indian river. What will be the correct pronunciation?
The first step in each FA-proces, is the transformation from words to phonemes. Then, these phonemes will be matched with the spoken content in the audio file. This means that the phonetic transcription is crucial but this transcription is language-dependent! E.g. the word boat will be transcribed as /b o: t/ when you use the English G2P (= Grapheme-to-Phoneme) converter. But if you use the Dutch G2P, the phonetic transcription will be /b o: A t/.
If we pass the example sentence in a G2P, it will result in:
Mrs Uytendenboogaard and Mr. Chukwuemeka have a boat trip on the Bhagirathi River
The transcription was done with an automatic G2P for English and Dutch. The blue words mean: "No idea how to pronounce it". The Dutch G2P decided to spell the unknown words (so Mr. becomes Em Er). The difference between these two transcriptions is huge and the Forced Alignements with the two transcription may significantly differ as well.
This small example shows that the phonetic transcription, crucial for correct Forced Alignment, is less straightforward than often expected. Abbreviations and unknown/foreign words may ruin the alignment. A good FA-system must therefore interact with the user about the correct phonetic transcription: presenting the user the best guess of initially unknown words. The user may enter the phonetic transcription of the words (using the phonetic alphabet) or a "sound-as" word ("Bhagirathi" sounds as "bakirati"). Once the initially unknown word is transcribed by the user, it may entered into a phonetic dictionary for further use.
The quality of the audio-signal (i.e. the speech sound) may influence the accuracy of the alignment. The better the audio (less or no background noise), the higher the accuracy.
A very useful service to align audio and text is the CLARIN WebMAUS-basic service of the Phonetic department of the LMU (München). The service needs two files: the audio-file and an ASCII/UTF-8 text file, containing the transcription.
TextGrid-file (a format you can use in PRAAT, see fig. 1 on the right) that contains both an orthographic output (the start and end times of the words in the transcription-file) and an phonetic output (the start and end times of the phonemes of the words).
For Oral History, the orthographic output is enough.
WebMAUS has more then 36 languages to choose from. In case a specific language is not available, the software works well with a language that is (acoustically) close to the spoken language. For example, we have forced aligned a Russian spoken interview with Dutch as the selected language ? .
There are a couple of issues when starting to align the transcriptions. The main issue are the "abbreviations": is the difference between the way a word is written and how it is pronounced by the majority of the native speakers.
Easy problems are words without a vowel: the are spelled out or "replaced" by the original word.
- NCRV → /E n - s e - E r - v e/ (A Ducth broadcast organisation)
- Mr → Mister → /M I s - t @ r/
If words contain one or more vowels, it depends on the local attitudes.
- NOS → N. O. S. → /E n - o - E s/ (the national Dutch broadcast organisation). NOS can be pronounced as nos /n O s/, but no one does it.
- RAI→ /r Ai/ (the national Italian broadcast organisation). The word is perfectly pronouncable in Italian, so no one is using the spell-mode.
A particular kind of abbreviations are those that depends of the context.
- An appointment with dr. Corti → an appointment with doctor Corti
- An appointment on the Corti dr. → an appointment on the Corti drive
A special kind of abbreviations are numbers. A normal number like 19 → sounds as nineteen → /n Ai n - t i n/. But numbers are context sensitive as well.
- my phone number is 621888146 → 6 2 1 8 8 8 (or 3 times 8) 1 4 6
- the cost in euro of that bridge is 621888146 → sixhundred twenty one million etc.
So, before using the G2P, one needs to preprocess the text in order to know what the most likely way of pronouncation will be.
When a transcription is made by experienced transcribers, one may expect that these kind of "problems" are (partly) solved by the way the speech is orthographically transcribed. So if someone say "the doctor Luther King drive" you may hope that it is not transcribed as "the dr. Luther King dr.". In general, it is preferable when transcribers do not use abbreviations or numbers, but for practical reasons (speed, out of habit) they do ?.
So, transcribers will probably use numbers as shows the transcription example in fig. 3 below. In order to align the text with the audio however, WebMAUS first rewrites abbreviations and numbers into the full-words (Mr → Mister, 19 → nineteen). This means that the transcription you present and the resulting TextGrid-file differ on abbreviations and numbers.
[Input WebMAUS] "Mr Arjan is 19 years old" → [Output WebMAUS] "Mister Arjan is nineteen years old".
Another issue is the speaker segmentation. In many human-made transcriptions, the speakers are annotated with their name or role, followed by a column ("John: I was working in London, when....." or "Int: I was working in London, when...")
The words John or Int are not spoken, so they have to be removed before aligning the audio with the text. But this means that all information about who-said-what, is removed as well.
So, the end-result of forced alignment by WebMAUS, is one string of words, independent of who was speaking and without dots, question marks, commas and other punctuations.
Cleaning the transcription
The original transcription (aka the word-file), contains all the required information: the words, the speakers, the punctuations and the non-verbal utterances as can be seen in the screenshots of the English (Fig. 3 above) and Italian (Fig. 7 below) transcripts.The required input for WebMAUS, is a plain ASCII/UTF-8 text file without all information other than the spoken words. So, the first step is to save the MS-word transcription files in Text-only UTF-8. The easiest way to do so, is to do it by hand from within the Word-application (save-as..).
Once the UTF-8 files are available, the can be read with Transcription Cleaner, a small Windows/OSX application that allows the user to select the transcription text and saves it in a XML-file with turn's containing the speaker as an attribute and the <original> and <cleaned> text as two text-elements (see Fig. 5 below).
Besides this XML-file, the program saves the cleaned text-elements as an UTF-8 text-file with just the cleaned text. This cleaned text-file can be used for the WebMAUS forced alignment.
Aligning the FA-result with the non-cleaned transcription
The results of the Forced Alignment (FA) by WebMAUS is a TextGrid-file (as can be seen in Fig. 1 above) with just the aligned words with their start and end-times. In another program these WebMAUS results and the cleaned text are 'aligned' resulting in an aligment result that includes the speaker-ID. The next months we will try to clue the WebMAUS-results not only to the cleaned text, but to the original text. If this can be done, it will result in the original transcription including the beginning and end of each transcribed word.
We have tried to do so for some Italian transcripts. We succeed due to the fact that in the original Italian transcriptions were rather strait-forward and did not contain abbreviations or numbers (for the explanation, see below) .
In fig 6 we see the original transcription and in fig. 7 the final result: a player that highlites the word spoken.
The software works well but there are some "issues". In the English transcriptions, ofte number are used (for example: "I saw 2 ladies of 36 years").
WebMAUS internally rewrites 2 → two and 36 → thirtysix. This is done because WebMAUS is basically a phonetic aligner and thus needs the numbers to be spelled out in order to do this "phonetic alignment". The same is true for abbreviations: mr. → mister etc.
But as a result, the application has to align 36 with thirtysix, and mr. to mister which makes the software much more complex.