TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizers
Abstract
Neural codec language model (LM) has demonstrated strong capability in zero-shot text-to-speech (TTS) synthesis. However, the codec LM often suffers from limitations in inference speed and stability, due to its auto-regressive nature and implicit alignment between text and audio. In this work, to handle these challenges, we introduce a new variant of neural codec LM, namely TacoLM. Specifically, TacoLM introduces a gated attention mechanism to improve the training and inference efficiency and reduce the model size. Meanwhile, an additional gated cross-attention layer is included for each decoder layer, which improves the efficiency and content accuracy of the synthesized speech. In the evaluation of the Librispeech corpus, the proposed TacoLM achieves a better word error rate, speaker similarity, and mean opinion score, with 90% fewer parameters and 5.2 times speed up, compared with VALL-E. Demo and code is available at https://anonymous.4open.science/w/TacoLM
[*] This page is for research demonstration purposes only.
Zero-Shot TTS Continutaion Samples.
We perform the zero-shot continuation task on the LibriSpeech test-clean set. For each sample, we take the first 3 seconds as the audio prompt and make it generate the original text of the sentence (as the text prompt). For the experiments of YourTTS, we had it synthesize complete sentences rather than continuing from the speech prompt. 8 samples are randomly selected and shown below.
| Forthwith, all ran to the opening of the tent to see what might be amiss. But Master Will, who peeped out first, needed no more than one glance. | ||||||
| The undertaking, which seemed so attractive when viewed as a literary task, proved to be a most difficult one, and at times, it became oppressive. | ||||||
| Many laws exist among us which are the counterpart of yours as they were in the olden time. | ||||||
| I will briefly describe them to you, and you shall read the account of them at your leisure in the sacred registers. | ||||||
| Again, I thank you. This incident, I suppose, will be renewed no more. If I live to be an old woman, I shall remember it thirty years hence as a bright dream. | ||||||
| She saw that the bed was gilded and so rich that it seemed that of a prince rather than of a private gentleman. | ||||||
| Some points may be taken as fixed, and such as any theory of memory must arrive at. | ||||||
| Then she gave Rosalie back her magic ring, thanking the kind witch for all she had done for them. |