Nice, that's a good solution. And yeah, I found out this morning that it picks up from where it left off very well. How many epochs did you leave it to train for? I'm on 100k so far and will probably let it run until close to a million I guess.

I just followed in this guy's steps and fine-tuned the pre-trained model on my own data. I tried going up to 300k, but I found it starts getting worse after ~260k. I don't think I ever tried training it from scratch.

1 million epochs? Wow, that would take quite a while on my hardware. Can I ask what GPU you're using and how fast your training goes?

Nice, that's a good solution. And yeah, I found out this morning that it picks up from where it left off very well. How many epochs did you leave it to train for? I'm on 100k so far and will probably let it run until close to a million I guess.

I'm seeing the same thing. Did you find a fix?

I didn't find a fix but I did find a workaround: Automatically restarting after a crash.

train.bat:

:loop
python train_tacotron.py
echo Crash detected, restarting...
timeout /t 5 /nobreak
goto loop

Also, are you able to pick up training from where you left off?

Yep, it always restarts from the latest step for me.

@EuphoriaCelestial, I've installed successfully, how many hours (minimum) of audio need to generate correct spelled audio ?

just comment out the line, you can add it back the next time to generate a checkpoint.

Unable to run train.py : File "train.py", line 188, in <module> train(num_gpus, args.rank, args.group_name, **train_config) File "train.py", line 90, in train optimizer) File "train.py", line 45, in load_checkpoint optimizer.load_state_dict(checkpoint_dict['optimizer']) File "C:\Python37\lib\site-packages\torch\optim\optimizer.py", line 124, in load_state_dict raise ValueError("loaded state dict contains a parameter group " ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group

Can you help me, please?

I realise each case can be different but I'm pretty sure it went okay for me using the WaveGrad in TTS with the LR at 1e-4, although I did hit some minor problems where I was continuing training (although looks like those aren't quite what you have here).

For the original error, "CUDA error: device-side assert triggered", I think you may be able to get more detail if you run it again with: CUDA_LAUNCH_BLOCKING="1"

I am trying to run my code on collab but it gives me this error.

Traceback (most recent call last):
  File "TTS/train.py", line 27, in <module>
    from TTS.utils.speakers import load_speaker_mapping, save_speaker_mapping, \
  File "/usr/local/lib/python3.6/dist-packages/TTS-0.0.3+b1935c9-py3.6.egg/TTS/utils/speakers.py", line 4, in <module>
    from TTS.datasets.preprocess import get_preprocessor_by_name
  File "/usr/local/lib/python3.6/dist-packages/TTS-0.0.3+b1935c9-py3.6.egg/TTS/datasets/preprocess.py", line 9, in <module>
    from TTS.tts.utils.generic_utils import split_dataset
ModuleNotFoundError: No module named 'TTS.tts'

how to fix it?

That would be the most obvious, but it tried decreasing batchsizes from 96 (default) down to 8 with no effect. I also tried changing lr from 1e-4 (default) to 2e-4 (default value in freds0 wavegrad implementation).

I initially set "stats_path" to the "taco2-scale_stats.npy" from our taco2 model training (is this a good idea in general)? If i leave "stats_path" empty i receive a warning before training is failing with previous written error:

 > TRAINING (2020-11-30 15:32:13) 
/home/thorsten/___prj/tts/models/vocoder/wavegrad/mozilla/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)

it is probably OOM error in a different face.

Error when I try to train vocoder with custom dataset.

 > Using CUDA:  True
 > Number of GPUs:  1
fatal: not a git repository (or any of the parent directories): .git
 > Git Hash: 0000000
 > Experiment folder: vocoder_model/multiband-melgan-Kasım-26-2020_01+29ÖS-0000000
fatal: not a git repository (or any of the parent directories): .git
 > Loading wavs from: LJSpeech-1.1/wavs/
 > Setting up Audio Processor...
 | > sample_rate:22050
 | > num_mels:80
 | > min_level_db:-100
 | > frame_shift_ms:None
 | > frame_length_ms:None
 | > ref_level_db:0
 | > fft_size:1024
 | > power:None
 | > preemphasis:0.0
 | > griffin_lim_iters:None
 | > signal_norm:True
 | > symmetric_norm:True
 | > mel_fmin:50.0
 | > mel_fmax:7600.0
 | > spec_gain:1.0
 | > stft_pad_mode:reflect
 | > max_norm:4.0
 | > clip_norm:True
 | > do_trim_silence:True
 | > trim_db:60
 | > do_sound_norm:False
 | > stats_path:scale_stats.npy
 | > hop_length:256
 | > win_length:1024
 > Generator Model: multiband_melgan_generator
 > Discriminator Model: melgan_multiscale_discriminator
 > Generator has 2539448 parameters
 > Discriminator has 4354998 parameters

 > EPOCH: 0/10000

 > TRAINING (2020-11-26 13:29:50) 
 ! Run is removed from vocoder_model/multiband-melgan-Kasım-26-2020_01+29ÖS-0000000
Traceback (most recent call last):
  File "TTS/vocoder/train.py", line 648, in <module>
    main(args)
  File "TTS/vocoder/train.py", line 552, in main
    epoch)
  File "TTS/vocoder/train.py", line 101, in train
    for num_iter, data in enumerate(data_loader):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 971, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1014, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
ValueError: Caught ValueError in DataLoader worker process 2.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/TTS-0.0.3+b1935c9-py3.6.egg/TTS/vocoder/datasets/gan_dataset.py", line 70, in __getitem__
    item1 = self.load_item(idx)
  File "/usr/local/lib/python3.6/dist-packages/TTS-0.0.3+b1935c9-py3.6.egg/TTS/vocoder/datasets/gan_dataset.py", line 116, in load_item
    mel_start = random.randint(0, max_mel_start)
  File "/usr/lib/python3.6/random.py", line 221, in randint
    return self.randrange(a, b+1)
  File "/usr/lib/python3.6/random.py", line 199, in randrange
    raise ValueError("empty range for randrange() (%d,%d, %d)" % (istart, istop, width))
ValueError: empty range for randrange() (0,-8, -8)

How to fix it?

probably.

I close this issue as you fixed it.

First things first: Thanks @nmstoker for providing great gatherup tool :+1: .

• I'm trying to train WaveGrad vocoder model for use with our pretrained taco2 thorsten model (ljspeech structure)
• Using NVidia xavier AGX machine (ubuntu aarch64) for training
• Copied and adjusted "wavegrad_libritts.json" to match our taco2 audio settings
• I've uploaded taco2 config (irrelevant for wavegrad training, or?) and vocoder config used for training. wavegrad-thorsten-conf.zip

Details

While in epoch 0 i get following error:

python ./TTS/bin/train_vocoder_wavegrad.py --config_path ./TTS/vocoder/configs/wavegrad_thorsten.json 
 > Using CUDA:  True
 > Number of GPUs:  1
   >  Mixed precision is enabled
 > Git Hash: ac46c3f
 > Experiment folder: /home/thorsten/___prj/tts/models/vocoder/wavegrad/mozilla/wavegrad-model-output/wavegrad-thorsten-November-30-2020_01+03PM-ac46c3f
 > Loading wavs from: /home/thorsten/___prj/tts/datasets/thorsten-de_v02/
 > Setting up Audio Processor...
 | > sample_rate:22050
 | > resample:False
 | > num_mels:80
 | > min_level_db:-100
 | > frame_shift_ms:None
 | > frame_length_ms:None
 | > ref_level_db:20
 | > fft_size:1024
 | > power:None
 | > preemphasis:0.0
 | > griffin_lim_iters:None
 | > signal_norm:True
 | > symmetric_norm:True
 | > mel_fmin:0
 | > mel_fmax:8000.0
 | > spec_gain:20.0
 | > stft_pad_mode:reflect
 | > max_norm:1.0
 | > clip_norm:True
 | > do_trim_silence:True
 | > trim_db:60
 | > do_sound_norm:False
 | > stats_path:/home/thorsten/___prj/tts/models/vocoder/wavegrad/mozilla/taco2-files/taco2-scale_stats.npy
 | > hop_length:256
 | > win_length:1024
 > Generator Model: wavegrad
 > WaveGrad has 15827106 parameters

 > EPOCH: 0/10000

 > TRAINING (2020-11-30 13:04:15) 
/media/nvidia/WD_NVME/PyTorch/JetPack_4.4/GA/pytorch-v1.6.0/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [0,0,0], thread: [47,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
 ! Run is removed from /home/thorsten/___prj/tts/models/vocoder/wavegrad/mozilla/wavegrad-model-output/wavegrad-thorsten-November-30-2020_01+03PM-ac46c3f
Traceback (most recent call last):
  File "./TTS/bin/train_vocoder_wavegrad.py", line 504, in <module>
    main(args)
  File "./TTS/bin/train_vocoder_wavegrad.py", line 401, in main
    epoch)
  File "./TTS/bin/train_vocoder_wavegrad.py", line 116, in train
    noise, x_noisy, noise_scale = model.compute_y_n(x)
  File "/home/thorsten/___prj/tts/models/vocoder/wavegrad/mozilla/lib/python3.6/site-packages/TTS-0.0.7+ac46c3f-py3.6-linux-aarch64.egg/TTS/vocoder/models/wavegrad.py", line 110, in compute_y_n
    noise_scale = l_a + torch.rand(y_0.shape[0]).to(y_0) * (l_b - l_a)
RuntimeError: CUDA error: device-side assert triggered
(mozilla) thorsten@nvidia-agx:~/___prj/tts/models/vocoder/wavegrad/mozilla/TTS$ 

Platform OS

• Linux

Python Environment

• Python 3.6.9

• Virtual env: Venv / virtualenv

Package Installation

• TTS installed from source on GitHub <br> <details> <summary>Click to see package list (package count: 116)</summary><br>

  :package: Package list from Pip

  Package	Version
  absl-py	0.11.0
  astroid	2.4.2
  astunparse	1.6.3
  attrdict	2.0.1
  attrs	20.3.0
  audioread	2.1.9
  bokeh	1.4.0
  cachetools	4.1.1
  cardboardlint	1.3.0
  certifi	2020.11.8
  cffi	1.14.4
  chardet	3.0.4
  click	7.1.2
  clldutils	3.5.4
  colorama	0.4.4
  colorlog	4.6.2
  commonmark	0.9.1
  confuse	1.3.0
  csvw	1.8.1
  cycler	0.10.0
  Cython	0.29.21
  dataclasses	0.7
  decorator	4.4.2
  docopt	0.6.2
  filelock	3.0.12
  Flask	1.1.2
  future	0.18.2
  gast	0.3.3
  gatherup	0.0.4
  gdown	3.12.2
  german-transliterate	0.1.3
  google-auth	1.23.0
  google-auth-oauthlib	0.4.2
  google-pasta	0.2.0
  grpcio	1.33.2
  h5py	2.10.0
  idna	2.10
  importlib-metadata	3.1.0
  importlib-resources	3.3.0
  inflect	5.0.2
  isodate	0.6.0
  isort	4.3.21
  itsdangerous	1.1.0
  Jinja2	2.11.2
  joblib	0.17.0
  Keras-Preprocessing	1.1.2
  kiwisolver	1.3.1
  lazy-object-proxy	1.4.3
  librosa	0.7.2
  llvmlite	0.31.0
  Markdown	3.3.3
  MarkupSafe	1.1.1
  matplotlib	3.3.3
  mccabe	0.6.1
  nose	1.3.7
  num2words	0.5.10
  numba	0.48.0
  numpy	1.18.5
  oauthlib	3.1.0
  opt-einsum	3.3.0
  packaging	20.7
  phonemizer	2.2.1
  Pillow	8.0.1
  pip	20.2.4
  pkg-resources	0.0.0
  prompt-toolkit	3.0.8
  protobuf	3.14.0
  pyasn1	0.4.8
  pyasn1-modules	0.2.8
  pycparser	2.20
  Pygments	2.7.2
  pylint	2.5.3
  pyparsing	2.4.7
  pysbd	0.3.3
  PySocks	1.7.1
  python-dateutil	2.8.1
  pyworld	0.2.12
  PyYAML	5.3.1
  questionary	1.8.1
  regex	2020.11.13
  requests	2.25.0
  requests-oauthlib	1.3.0
  resampy	0.2.2
  rfc3986	1.4.0
  rich	8.0.0
  rsa	4.6
  scikit-learn	0.23.2
  scipy	1.4.1
  segments	2.1.3
  setuptools	50.3.2
  six	1.15.0
  SoundFile	0.10.3.post1
  tabulate	0.8.7
  tensorboard	2.4.0
  tensorboard-plugin-wit	1.7.0
  tensorboardX	2.1
  tensorflow	2.3.0+nv20.9
  tensorflow-estimator	2.3.0
  termcolor	1.1.0
  threadpoolctl	2.1.0
  toml	0.10.2
  torch	1.6.0
  tornado	6.1
  tqdm	4.54.0
  TTS	0.0.7+ac46c3f
  typed-ast	1.4.1
  typing-extensions	3.7.4.3
  umap-learn	0.4.6
  Unidecode	0.4.20
  uritemplate	3.0.1
  urllib3	1.26.2
  wcwidth	0.2.5
  Werkzeug	1.0.1
  wheel	0.35.1
  wrapt	1.12.1
  zipp	3.4.0

</details>

- generated at 13:31 on Nov 30 2020 using Gather Up tool :gift:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discourse page for further help. https://discourse.mozilla.org/c/tts

I think I fixed this issue by changing do_trimming argument as false in config file. Will it affect performance of model?

Hello, author! Do you have the download address for thewaveglow_old.pt model? I want to use this model for testing.