KARATUNER: TOWARDS END TO END NATURAL PITCH CORRECTION FOR SINGING VOICE IN KARAOKE
Abstract
An automatic pitch correction system typically includes several stages, such as pitch extraction, deviation estimation, pitch shift processing, and cross-fade smoothing. However, designing these components with strategies often requires domain expertise and they are likely to fail on corner cases. In this paper, we present KaraTuner, an end-to-end neural architecture that predicts pitch curve and resynthesizes the singing voice directly from the tuned pitch and vocal spectrum extracted from the original recordings. Several vital technical points have been introduced in KaraTuner to ensure pitch accuracy, pitch naturalness, timbre consistency, and sound quality. A feed-forward Transformer is employed in the pitch predictor to capture long-term dependencies in the vocal spectrum and musical note. We also develop a pitch-controllable vocoder base on a novel source-filter block and the Fre-GAN architecture. KaraTuner obtains a higher preference than the rule-based pitch correction approach through A/B tests, and perceptual experiments show that the proposed vocoder achieves significant advantages in timbre consistency and sound quality compared with the parametric WORLD vocoder and phase vocoder.
Architecture
Pitch Shift Examples
The song examples we list here are the original recordings, the outputs of KaraTuner, and the outputs of WORLD vocoder when shifting the pitch with three constant ratios: 0.84, 1.00, 1.19.
sample 1
| ratio | 0.84 | 1.0 | 1.19 |
| WORLD Vocoder | |||
| KaraTuner(proposed) |
sample 2
| ratio | 0.84 | 1.0 | 1.19 |
| WORLD Vocoder | |||
| KaraTuner(proposed) |
sample 3
| ratio | 0.84 | 1.0 | 1.19 |
| WORLD Vocoder | |||
| KaraTuner(proposed) |
sample 4
| ratio | 0.84 | 1.0 | 1.19 |
| WORLD Vocoder | |||
| KaraTuner(proposed) |
Pitch Correction Examples
The song examples we list here are the original recordings, the outputs of KaraTuner, the outputs of WORLD vocoder; the output of phase vocoder.
| Original Recording | KaraTuner(proposed) | WORLD Vocoder | Phase Vocoder |
Pitch Predictor Examples
The song examples we list here are the original recordings, the outputs of the baseline tuning method (post tuning process in A Neural Parametric Singing Synthesizer, NPSS) and the outputs of KaraTuner.
| Original Recording | KaraTuner(proposed) | NPSS |