Source, filter and aperiodicity

In the first-order source-filter picture a voice has a source that is either a buzz (voiced) or a hiss (unvoiced), and everything else is the filter: the vocal tract's formants. A vocoder such as WORLD (Morise, Yokomori & Ozawa, 2016) keeps three things instead of two: the pitch, a smooth spectral envelope, and an aperiodicity. The aperiodicity replaces the voicing switch. At every frequency it says what share of the power is noise rather than harmonics,

A(f) = \frac{N(f)}{N(f) + P(f)},

with N the power of the noise and P that of the harmonics at f. The buzz is A = 0 everywhere and the hiss is A = 1; a real voice is in between, and usually more so at high frequencies, so it can be clearly periodic at 500 Hz and mostly noise at 5 kHz.

Press play and a line follows the sound across every time axis in its plots. Click any time axis to play from that point. Start with your volume low.

The vowel, and code the examples share

The vowel of "hod", made as on the Formant synthesis page: Klatt's (1980) glottal source, the radiation from the lips, and five formants. Here the source is a sum of harmonics of a steady 115 Hz plus white noise, both put through the same formants. The source's harmonics fall about 6 dB per octave and the noise is flat, so the share of noise rises about 6 dB per octave. Its level is set so that noise and harmonics are equally strong at 4 kHz. Everything about the vowel is known, so its aperiodicity can be written down.

Code
import matplotlib.pyplot as plt
import numpy as np

import sonore as so

plt.rcParams.update({"font.size": 9, "axes.titlesize": 10, "figure.dpi": 100})
FS = 16000
F0 = 115.0
DUR = 1.0
HOD = [(730, 60), (1090, 90), (2440, 150), (3500, 200), (4500, 250)]


def finish(snd):
    """How every sound in the gallery is played: 5 ms ramps, RMS 0.1, peak at most 0.95."""
    snd = snd.ramp(5e-3).normalize(rms=0.1)
    return snd.normalize(peak=0.95) if snd.peak > 0.95 else snd


def show(snd, title, fmax=5000):
    """Waveform and narrowband spectrogram (Hann 33 ms, harmonics resolved), on one time axis."""
    fig = plt.figure(figsize=(10, 4.2), layout="constrained")
    ax0, ax1 = fig.subplots(2, 1, sharex=True, height_ratios=[0.45, 1])
    snd.plot(ax0, color="k", lw=0.4)
    ax0.set(title=f"Waveform: {title}", xlabel="")
    so.STFT(snd, win_dur=0.0333, hop_dur=0.002).plot(ax1, db_range=70, colorbar=False, fmax=fmax)
    ax1.set_title("Spectrogram (Hann 33 ms)")
    for ax in (ax0, ax1):
        ax.set_xlim(0, snd.duration)
    return fig, [ax0, ax1]


def formants(snd):
    for freq, bandwidth in HOD:
        snd = so.resonator(snd, freq, bandwidth)
    return snd


# Klatt's glottal source (an impulse through a low-pass resonator at 0 Hz, 100 Hz wide) and the
# radiation from the lips (a first difference), as a gain at any frequency.
impulse = so.Sound(np.r_[1.0, np.zeros(2**14 - 1)], FS)
glottal_gain = np.abs(np.fft.rfft(so.resonator(impulse, 0, 100).data[:, 0]))
gain_freqs = np.fft.rfftfreq(2**14, 1 / FS)


def source_gain(f):
    return np.interp(f, gain_freqs, glottal_gain) * np.abs(2 * np.sin(np.pi * np.asarray(f) / FS))


harmonic_numbers = np.arange(1, int(0.45 * FS / F0) + 1)
buzz = so.harmonic_complex(DUR, FS, F0, harmonics=harmonic_numbers, amplitudes=lambda t, f: source_gain(f))
# harmonic_complex scales its output to RMS 1; the harmonics' power per hertz near f is then
# (scale * gain(f))^2 / 2 per harmonic, one harmonic every F0 hertz
scale = 1 / np.sqrt(np.sum(source_gain(harmonic_numbers * F0) ** 2) / 2)


def harmonic_density(f):
    return (scale * source_gain(f)) ** 2 / 2 / F0


noise_density = harmonic_density(4000.0)  # noise as strong as the harmonics at 4 kHz
hiss = so.gaussian_noise(DUR, FS, rng=1) * np.sqrt(noise_density * FS / 2)  # white: flat density


def true_aperiodicity(f):
    """The share of noise at f, from the source alone."""
    return noise_density / (noise_density + harmonic_density(f))


vowel = formants(buzz + hiss)
window_times = np.arange(0, DUR, 0.005)
track = (window_times, np.full(len(window_times), F0))  # the exact F0, every 5 ms
inside = (window_times > 0.1) & (window_times < 0.9)  # time windows away from the ends

Buzz, hiss, and both

The first-order picture has two settings of the source, and the voice in between.

The buzz alone

The buzz alone: harmonics through the formants, A = 0 everywhere. Every frequency is periodic: the spectrogram shows harmonics all the way up, with nothing between them.

ap1_the_buzz_alone.flac, 1.0 s, mono

Plots of the the buzz alone sound
Code
sound = finish(formants(buzz))
fig, playhead = show(sound, "harmonics through the formants (A = 0)")

The hiss alone

The hiss alone: white noise through the same formants, A = 1 everywhere. The vowel is still there, whispered, because the formants are.

ap2_the_hiss_alone.flac, 1.0 s, mono

Plots of the the hiss alone sound
Code
sound = finish(formants(hiss))
fig, playhead = show(sound, "noise through the formants (A = 1)")

A breathy vowel

Both together. Below about 1 kHz the harmonics stand well above the noise; above 3 kHz the noise fills in between them and then covers them. This is what makes a voice breathy: neither switch setting, but a mixture whose proportion changes with frequency.

ap3_a_breathy_vowel.flac, 1.0 s, mono

Plots of the a breathy vowel sound
Code
sound = finish(vowel)
fig, playhead = show(sound, "harmonics and noise through the formants")

Its aperiodicity is the noise's share of the source at each frequency. The harmonics and the noise both pass through the same filter H, so the share after it is

\frac{|H(f)|^2 N(f)}{|H(f)|^2 N(f) + |H(f)|^2 P(f)} = \frac{N(f)}{N(f) + P(f)}:

the filter cancels. The aperiodicity is a property of the source, measured on the output, and it can be changed without touching the formants, and the formants without touching it.

Aperiodicity of the vowel, from its source

Aperiodicity of the vowel, from its source
Code
freqs = np.linspace(50, 7200, 1000)
fig, ax = plt.subplots(figsize=(10, 3.2), layout="constrained")
ax.plot(freqs, 10 * np.log10(true_aperiodicity(freqs)), color="k", label="the breathy vowel")
ax.axhline(0, color="C3", ls="--", lw=1, label="the hiss (A = 1)")
ax.axhline(-40, color="C0", ls="--", lw=1, label="the buzz (A = 0, at the bottom of the plot)")
ax.set(xlabel="Frequency (Hz)", ylabel="Share of noise (dB)", xlim=(0, 7200), ylim=(-40, 3))
ax.set_title("Aperiodicity: noise power over total power at each frequency")
ax.legend(loc="lower right", fontsize=8)

Fit the harmonics, keep the rest

The definition suggests the measurement: find the best periodic sound, take it away, and see how much is left. If the pitch is known, the periodic part of a short stretch is a sum of harmonics whose phases follow the running phase \Phi(t) = 2\pi \int F_0, each with an amplitude and a phase to be found. That is a linear least-squares fit. Here is one time window, under a Hann window four periods long, with each harmonic also allowed an amplitude that changes linearly across the window.

Code
centre = int(0.5 * FS)  # the time window at 0.5 s
half = int(round(2 * FS / F0))  # half of four periods
offsets = np.arange(-half, half + 1)
window = 0.5 + 0.5 * np.cos(np.pi * offsets / (half + 1))
phase = 2 * np.pi * F0 * offsets / FS  # the running phase over the time window
harmonic_phases = np.outer(phase, np.arange(1, int(FS / 2 / F0) + 1))
ramp = (offsets / half)[:, None]
columns = np.column_stack(
    [np.ones(len(offsets)), np.cos(harmonic_phases), np.sin(harmonic_phases)]
    + [ramp * np.cos(harmonic_phases), ramp * np.sin(harmonic_phases)]
)
segment = vowel.data[centre + offsets, 0]
root_window = np.sqrt(window)  # weighted least squares: minimise the windowed residual
weights, *_ = np.linalg.lstsq(columns * root_window[:, None], segment * root_window, rcond=None)
residual = segment - columns @ weights
print(f"{columns.shape[1]} columns fitted to {len(offsets)} samples")
residual_power, signal_power = np.sum((window * residual) ** 2), np.sum((window * segment) ** 2)
print(f"windowed residual power over windowed signal power: {residual_power / signal_power:.3f}")
277 columns fitted to 557 samples
windowed residual power over windowed signal power: 0.012

One time window, before and after the harmonics are taken away

The time window's spectrum, and the spectrum of what is left after the fit. The harmonic peaks are gone from the residual, and what remains is the noise between and under them. Where the noise was below the harmonics (low frequencies) the residual lies far below the time window's spectrum; above 4 kHz the two are close. Their ratio, read across frequency, is the aperiodicity.

One time window, before and after the harmonics are taken away
Code
n_fft = 4096
segment_freqs = np.fft.rfftfreq(n_fft, 1 / FS)
segment_db = 20 * np.log10(np.abs(np.fft.rfft(window * segment, n_fft)) + 1e-12)
residual_db = 20 * np.log10(np.abs(np.fft.rfft(window * residual, n_fft)) + 1e-12)
fig, ax = plt.subplots(figsize=(10, 3.4), layout="constrained")
ax.plot(segment_freqs, segment_db, color="0.6", lw=0.7, label="the time window")
ax.plot(segment_freqs, residual_db, color="C3", lw=0.7, label="the time window less the fitted harmonics")
top = segment_db.max() + 5
ax.set(xlabel="Frequency (Hz)", ylabel="Level (dB)", xlim=(0, 6000), ylim=(top - 95, top))
ax.set_title("One time window of the breathy vowel (Hann, four periods)")
ax.legend(loc="upper right", fontsize=8)

so.harmonic_aperiodicity does this at every time window. It adds one correction: the fit also absorbs a little of the noise, the part that happens to look like the harmonics near each harmonic frequency, so the residual is a little too small there. How much is known exactly from the fit itself (it is what the fit would do to white noise), and is divided out. The shares are then summed over cells two harmonics wide and put on the frequency grid of WORLD's envelope.

The measured share of noise against the truth

The measured share of noise against the truth
Code
residual_share = so.harmonic_aperiodicity(vowel, track)
fig, ax = plt.subplots(figsize=(10, 3.2), layout="constrained")
ax.plot(freqs, 10 * np.log10(true_aperiodicity(freqs)), color="k", lw=2.5, alpha=0.35, label="truth")
measured = residual_share.share[0][:, inside]
ax.plot(residual_share.f, 10 * np.log10(measured.mean(axis=1)), color="C3", label="so.harmonic_aperiodicity")
ax.set(xlabel="Frequency (Hz)", ylabel="Share of noise (dB)", xlim=(0, 7200), ylim=(-40, 3))
ax.set_title("The harmonic residual reads the vowel's aperiodicity (mean over time windows)")
ax.legend(loc="lower right", fontsize=8)

The measure is the definition, which makes it easy to explain and to check. It has one weakness: it needs the pitch to about a tenth of a percent. A harmonic whose frequency is slightly wrong drifts out of phase with the fit over the window, and what the fit misses reads as noise, more so the higher the harmonic. For the same reason it reads cycle-to-cycle irregularity (jitter and shimmer) as noise.

What WORLD reports

WORLD's own measure is D4C (Morise, 2016), which so.d4c reproduces exactly. It works from a "group delay" of the time window, smoothed over the harmonics, whose spectrum is sorted to see how much of its power lies outside the strongest components, in bands 3 kHz wide around each multiple of 3 kHz. A correction for F0 follows, and the curve is drawn as straight lines (in dB) from −60 dB at 0 Hz through the bands' values to 0 dB at the top. At 16 kHz there is only one band, at 3 kHz. D4C's paper says it was tuned with listening tests so that WORLD's resynthesis sounds natural; the −60 dB at 0 Hz says "the lowest frequencies are periodic", which is usually true of speech.

D4C and the harmonic residual against the truth

D4C and the harmonic residual against the truth
Code
d4c_share = so.d4c(vowel, track)
fig, ax = plt.subplots(figsize=(10, 3.2), layout="constrained")
ax.plot(freqs, 10 * np.log10(true_aperiodicity(freqs)), color="k", lw=2.5, alpha=0.35, label="truth")
ax.plot(residual_share.f, 10 * np.log10(measured.mean(axis=1)), color="C3", label="so.harmonic_aperiodicity")
d4c_mean = d4c_share.share[0][:, inside].mean(axis=1)
ax.plot(d4c_share.f, 10 * np.log10(d4c_mean), color="C0", label="so.d4c (WORLD)")
ax.set(xlabel="Frequency (Hz)", ylabel="Share of noise (dB)", xlim=(0, 7200), ylim=(-62, 3))
ax.set_title("D4C at 16 kHz: one measured value at 3 kHz, joined to −60 dB at 0 Hz and 0 dB at 8 kHz")
ax.legend(loc="lower right", fontsize=8)

So the two answer different questions. The harmonic residual reports the share of noise. D4C reports a value that makes WORLD's synthesis sound natural, and below 3 kHz it is far from the share of noise on this vowel. In exchange, D4C hardly cares whether the pitch is exact. Averaged over bands and time windows, with the exact pitch track and with one 1% too high:

Code
band_edges = [0, 1000, 2000, 4000, 7000]
envelope = so.cheaptrick(vowel, track)
# the truth in each band: noise power over total power, the envelope weighting each frequency
# as .bands() weights it
weights = envelope.data[0][:, inside].mean(axis=1)
truth_db = []
for low, high in zip(band_edges[:-1], band_edges[1:], strict=True):
    in_band = (envelope.f >= low) & (envelope.f < high)
    share = np.sum(true_aperiodicity(envelope.f[in_band]) * weights[in_band]) / np.sum(weights[in_band])
    truth_db.append(10 * np.log10(share))
wrong_track = (window_times, track[1] * 1.01)
bands = "".join(f"{f'{low}-{high}':>11}" for low, high in zip(band_edges[:-1], band_edges[1:], strict=True))
print(f"{'Share of noise [dB], band [Hz]':<34}{bands}")
print(f"{'truth':<34}" + "".join(f"{value:>11.1f}" for value in truth_db))
for name, measure in [("harmonic residual", so.harmonic_aperiodicity), ("D4C", so.d4c)]:
    for label, f0 in [("exact F0", track), ("F0 1% high", wrong_track)]:
        shares = measure(vowel, f0).bands(band_edges, envelope)[0][:, inside].mean(axis=1)
        print(f"{name + ', ' + label:<34}" + "".join(f"{value:>11.1f}" for value in 10 * np.log10(shares)))
Share of noise [dB], band [Hz]         0-1000  1000-2000  2000-4000  4000-7000
truth                                   -13.7      -10.8       -5.2       -2.8
harmonic residual, exact F0             -13.4      -10.4       -5.5       -3.1
harmonic residual, F0 1% high           -13.4       -9.3       -2.9       -0.2
D4C, exact F0                           -47.5      -40.8      -15.1       -5.7
D4C, F0 1% high                         -47.4      -40.8      -15.0       -5.7

With the wrong pitch, the harmonic residual's upper bands move toward 0 dB (all noise), as the high harmonics drift out of phase with the fit; D4C's numbers do not change.

A sentence

The sentence from Seeing speech, with the F0 track WORLD's Harvest (Morise, 2017) measured, stored with it. On recorded speech there is no truth to compare with, but the two maps can be set side by side. Both are 1 (0 dB, all noise) where the track says unvoiced. In voiced time windows the harmonic residual is low below 1 to 2 kHz, where the harmonics are strong, and near 0 dB above about 4 kHz in many time windows; D4C has the same shape in every voiced time window, a single bend at 3 kHz.

Code
sentence = finish(so.load("docs/speech/bdl_arctic_a0131.flac"))
harvest_times, harvest_f0 = np.loadtxt("docs/speech/bdl_arctic_a0131_f0.csv", delimiter=",", skiprows=2).T
harvest = (harvest_times, harvest_f0)
sentence_envelope = so.cheaptrick(sentence, harvest)
sentence_d4c = so.d4c(sentence, harvest)
sentence_residual = so.harmonic_aperiodicity(sentence, harvest)

Envelope and aperiodicities of the sentence

Envelope and aperiodicities of the sentence
Code
fig, axes = plt.subplots(3, 1, figsize=(10, 7.2), sharex=True, layout="constrained")
sentence_envelope.plot(axes[0], db_range=70)
axes[0].set_title("Spectral envelope (CheapTrick)")
for ax, aperiodicity, title in [
    (axes[1], sentence_d4c, "Aperiodicity: so.d4c (WORLD)"),
    (axes[2], sentence_residual, "Aperiodicity: so.harmonic_aperiodicity"),
]:
    aperiodicity.plot(ax, db_range=40)
    ax.set_title(title)
for ax in axes:
    ax.set_xlim(0, sentence.duration)
axes[2].set_xlabel("Time (s)")

Listening

so.world_synthesize is WORLD's synthesis, reproduced sample for sample. At every pitch period it adds two pieces: the harmonics' share of the envelope, S(1 - A), as one pulse, and the noise's share, S A, as a short burst of filtered noise. Below, the same pitch track and envelope with four aperiodicities. (WORLD keeps A between −60 dB and just under 0 dB, and time windows the track calls unvoiced are always noise.)

The sentence

The sentence, for reference.

ap9_the_sentence.flac, 2.5 s, mono

Plots of the the sentence sound
Code
sound = sentence
fig, playhead = show(sound, "the sentence", fmax=8000)

Resynthesis with D4C

WORLD's resynthesis, with D4C's aperiodicity: what WORLD itself makes from this pitch track, sample for sample.

ap10_resynthesis_with_d4c.flac, 2.5 s, mono

Plots of the resynthesis with d4c sound
Code
sound = finish(so.world_synthesize(harvest, sentence_envelope, sentence_d4c))
fig, playhead = show(sound, "so.world_synthesize with so.d4c", fmax=8000)

With the harmonic residual

The same with the harmonic residual's aperiodicity, which puts more of the voice into noise, mostly at high frequencies.

ap11_with_the_harmonic_residual.flac, 2.5 s, mono

Plots of the with the harmonic residual sound
Code
sound = finish(so.world_synthesize(harvest, sentence_envelope, sentence_residual))
fig, playhead = show(sound, "so.world_synthesize with so.harmonic_aperiodicity", fmax=8000)

No noise in the voice

With A = 0 in every voiced time window: harmonics only, the voicing switch of the first-order picture. The voiced parts are harmonics all the way up.

ap12_no_noise_in_the_voice.flac, 2.5 s, mono

Plots of the no noise in the voice sound
Code
periodic = so.Aperiodicity(np.zeros_like(sentence_d4c.data), sentence_d4c.t, FS, "none")
sound = finish(so.world_synthesize(harvest, sentence_envelope, periodic))
fig, playhead = show(sound, "so.world_synthesize with A = 0", fmax=8000)

Nothing but noise

With A = 1 everywhere: noise only, through the same envelope. The pitch is gone and the sentence is whispered; it is still intelligible, because the envelope carries the words.

ap13_nothing_but_noise.flac, 2.5 s, mono

Plots of the nothing but noise sound
Code
noise_only = so.Aperiodicity(np.ones_like(sentence_d4c.data), sentence_d4c.t, FS, "all noise")
sound = finish(so.world_synthesize(harvest, sentence_envelope, noise_only))
fig, playhead = show(sound, "so.world_synthesize with A = 1", fmax=8000)

A higher voice

Female voices sit about half an octave to an octave above male ones, so their harmonics are further apart, and both measures have fewer of them to work with. The breathy vowel again, made the same way but at 230 Hz, twice the pitch, with female formants for "hod" (average F1 to F3 from Hillenbrand et al., 1995). The noise is again as strong as the harmonics at 4 kHz, so the truth has the same shape as before.

Code
F0_HIGH = 230.0
HOD_WOMEN = [(936, 60), (1551, 90), (2815, 150), (4100, 200), (4900, 250)]
harmonic_numbers_high = np.arange(1, int(0.45 * FS / F0_HIGH) + 1)
buzz_high = so.harmonic_complex(
    DUR, FS, F0_HIGH, harmonics=harmonic_numbers_high, amplitudes=lambda t, f: source_gain(f)
)
scale_high = 1 / np.sqrt(np.sum(source_gain(harmonic_numbers_high * F0_HIGH) ** 2) / 2)


def harmonic_density_high(f):
    return (scale_high * source_gain(f)) ** 2 / 2 / F0_HIGH


noise_density_high = harmonic_density_high(4000.0)
hiss_high = so.gaussian_noise(DUR, FS, rng=1) * np.sqrt(noise_density_high * FS / 2)
vowel_high = buzz_high + hiss_high
for freq, bandwidth in HOD_WOMEN:
    vowel_high = so.resonator(vowel_high, freq, bandwidth)
track_high = (window_times, np.full(len(window_times), F0_HIGH))
residual_high = so.harmonic_aperiodicity(vowel_high, track_high)
d4c_high = so.d4c(vowel_high, track_high)
n_low, n_high = len(harmonic_numbers), len(harmonic_numbers_high)
print(f"harmonics below 7.2 kHz: {n_low} at {F0:.0f} Hz, {n_high} at {F0_HIGH:.0f} Hz")
harmonics below 7.2 kHz: 62 at 115 Hz, 31 at 230 Hz

Both measures at twice the pitch

Both measures at 115 Hz (thin lines, as above) and at 230 Hz (thick lines), against the truth, which is the same at both pitches. The harmonic residual fits half as many harmonics at the higher pitch, in a window four of its shorter periods long; D4C's single band at 3 kHz gets a different F0 correction.

Both measures at twice the pitch
Code
fig, ax = plt.subplots(figsize=(10, 3.4), layout="constrained")
ax.plot(freqs, 10 * np.log10(true_aperiodicity(freqs)), color="k", lw=2.5, alpha=0.35, label="truth")
for share, color in [(residual_share, "C3"), (d4c_share, "C0")]:
    ax.plot(share.f, 10 * np.log10(share.share[0][:, inside].mean(axis=1)), color=color, lw=0.8, alpha=0.6)
for share, color, label in [
    (residual_high, "C3", "so.harmonic_aperiodicity"),
    (d4c_high, "C0", "so.d4c (WORLD)"),
]:
    mean_share = share.share[0][:, inside].mean(axis=1)
    ax.plot(share.f, 10 * np.log10(mean_share), color=color, lw=2, label=f"{label}, {F0_HIGH:.0f} Hz")
ax.set(xlabel="Frequency (Hz)", ylabel="Share of noise (dB)", xlim=(0, 7200), ylim=(-62, 3))
ax.set_title(f"The breathy vowel at {F0:.0f} Hz (thin) and {F0_HIGH:.0f} Hz (thick)")
ax.legend(loc="lower right", fontsize=8)
Code
truth_high_db = []
envelope_high = so.cheaptrick(vowel_high, track_high)
weights_high = envelope_high.data[0][:, inside].mean(axis=1)
for low, high in zip(band_edges[:-1], band_edges[1:], strict=True):
    in_band = (envelope_high.f >= low) & (envelope_high.f < high)
    share = np.sum(true_aperiodicity(envelope_high.f[in_band]) * weights_high[in_band]) / np.sum(
        weights_high[in_band]
    )
    truth_high_db.append(10 * np.log10(share))
print(f"{'Share of noise [dB], band [Hz]':<34}{bands}")
print(f"{'truth':<34}" + "".join(f"{value:>11.1f}" for value in truth_high_db))
for name, measure in [("harmonic residual", so.harmonic_aperiodicity), ("D4C", so.d4c)]:
    shares = measure(vowel_high, track_high).bands(band_edges, envelope_high)[0][:, inside].mean(axis=1)
    print(
        f"{name + f', {F0_HIGH:.0f} Hz':<34}" + "".join(f"{value:>11.1f}" for value in 10 * np.log10(shares))
    )
Share of noise [dB], band [Hz]         0-1000  1000-2000  2000-4000  4000-7000
truth                                   -12.3       -9.7       -4.7       -2.8
harmonic residual, 230 Hz               -18.6      -11.5       -5.4       -3.5
D4C, 230 Hz                             -44.2      -35.5      -10.8       -5.3

The same sentence read by a female talker (CMU ARCTIC, speaker slt). There is no stored Harvest track for this recording, so the pitch comes from so.f0_track; the analysis is still WORLD's.

Code
sentence_female = finish(so.load("docs/speech/slt_arctic_a0131.flac"))
track_female = so.f0_track(sentence_female)
envelope_female = so.cheaptrick(sentence_female, track_female)
d4c_female = so.d4c(sentence_female, track_female)
residual_female = so.harmonic_aperiodicity(sentence_female, track_female)

Envelope and aperiodicities of a higher voice

The female talker's envelope and the two aperiodicities, drawn as for the male talker's sentence above.

Envelope and aperiodicities of a higher voice
Code
fig, axes = plt.subplots(3, 1, figsize=(10, 7.2), sharex=True, layout="constrained")
envelope_female.plot(axes[0], db_range=70)
axes[0].set_title("Spectral envelope (CheapTrick), slt")
for ax, aperiodicity, title in [
    (axes[1], d4c_female, "Aperiodicity: so.d4c (WORLD), slt"),
    (axes[2], residual_female, "Aperiodicity: so.harmonic_aperiodicity, slt"),
]:
    aperiodicity.plot(ax, db_range=40)
    ax.set_title(title)
for ax in axes:
    ax.set_xlim(0, sentence_female.duration)
axes[2].set_xlabel("Time (s)")

The sentence, a higher voice

The female talker's sentence, for reference.

ap16_the_sentence_a_higher_voice.flac, 2.6 s, mono

Plots of the the sentence, a higher voice sound
Code
sound = sentence_female
fig, playhead = show(sound, "the sentence read by slt", fmax=8000)

Resynthesis with D4C, a higher voice

WORLD's resynthesis of the female talker's sentence with D4C's aperiodicity, on so.f0_track's pitch.

ap17_resynthesis_with_d4c_a_higher_voice.flac, 2.6 s, mono

Plots of the resynthesis with d4c, a higher voice sound
Code
sound = finish(so.world_synthesize(track_female, envelope_female, d4c_female))
fig, playhead = show(sound, "so.world_synthesize with so.d4c, slt", fmax=8000)

What this page leaves out

  • WORLD's pitch tracker. Harvest is not part of sonore; its track is stored with the sentence. With so.f0_track instead, the analysis is still WORLD's but the numbers are not the ones WORLD would give, which so.DIFFERENCES_FROM_WORLD notes.
  • Phase. WORLD's synthesis gives every pulse a minimum phase, so the waveform within each period is not the original's. Its authors call minimum phase inappropriate for low-pitched speech, where differences of phase are easier to hear.
  • Jitter and shimmer. A pitch track smoothed over 5 ms time windows cannot follow cycle-to-cycle irregularity, so the harmonic residual counts it as noise, and a resynthesis can only carry it as noise.

References

  • Hillenbrand, Getty, Clark & Wheeler (1995). Acoustic characteristics of American English vowels. J. Acoust. Soc. Am. 97(5), 3099–3111. doi:10.1121/1.411872. Female formants for "hod".
  • Klatt (1980). Software for a cascade/parallel formant synthesizer. J. Acoust. Soc. Am. 67(3), 971–995. doi:10.1121/1.383940. The glottal source.
  • Kominek & Black (2004). The CMU Arctic speech databases. Proc. 5th ISCA Speech Synthesis Workshop, 223–224. ISCA Archive. The sentence, by speakers bdl and slt.
  • Morise (2015). CheapTrick, a spectral envelope estimator for high-quality speech synthesis. Speech Communication 67, 1–7. doi:10.1016/j.specom.2014.09.003. spectral_envelope.cheaptrick
  • Morise (2016). D4C, a band-aperiodicity estimator for high-quality speech synthesis. Speech Communication 84, 57–65. doi:10.1016/j.specom.2016.09.001. aperiodicity.d4c
  • Morise (2017). Harvest: a high-performance fundamental frequency estimator from speech signals. Proc. Interspeech 2017, 2321–2325. doi:10.21437/Interspeech.2017-68. The stored F0 track.
  • Morise, Yokomori & Ozawa (2016). WORLD: a vocoder-based high-quality speech synthesis system for real-time applications. IEICE Trans. Inf. & Syst. E99-D(7), 1877–1884. doi:10.1587/transinf.2015EDP7457. world.world_synthesize