[Grid] You must call wandb.init() before wandb.log()

🐛 Bug

I’m reopening #1356 because I’m getting this error running my code on grid.ai.

I am getting error:

wandb.errors.error.Error: You must call wandb.init() before wandb.log()

Please reproduce using the BoringModel

Not possible since colab has only one GPU, unlike grid.ai

To Reproduce

On grid.ai or multiple GPU machine, create a trainer with WandbLogger and do not specify an accelerator. Run with gpus=-1 and hit this error.

Despite https://github.com/PyTorchLightning/pytorch-lightning/pull/2029 the default is ddp_spawn, which triggers this error on grid.ai:

/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:52: UserWarning: You requested multiple GPUs but did not specify a backend, e.g. `Trainer(accelerator="dp"|"ddp"|"ddp2")`. Setting `accelerator="ddp_spawn"` for you.

Workaround:

In main, run

import wandb
wandb.init(project...)

(seems redudant and potentially dangerous/foot-gunny since you are already passing a WandbLogger to the trainer.

Make sure trainer has accelerator=ddp defined.

Expected behavior

wandb logger works when trainer is given WandbLogger, gpu=-1, and no accelerator is defined, nor is a duplicate wandb init needed to be called.

Environment

grid.ai

CUDA: - GPU: - Tesla M60 - Tesla M60 - available: True - version: 10.2
Packages: - numpy: 1.20.2 - pyTorch_debug: False - pyTorch_version: 1.8.1+cu102 - pytorch-lightning: 1.2.7 - tqdm: 4.60.0
System: - OS: Linux - architecture: - 64bit - - processor: x86_64 - python: 3.7.10 - version: #1 SMP Tue Mar 16 04:56:19 UTC 2021

Issue Analytics

State:
Created 2 years ago
Reactions:1
Comments:6 (6 by maintainers)

Top GitHub Comments

1reaction

awaelchlicommented, Apr 17, 2021

Thanks. I tried this and can see where the problem is. Do the following:

Remove the manual wandb.init call at the bottom
Replace wandb.log({"examples": ... }) with self.logger.experiment.log(...)

This should work:) I can see the audio samples in the wandb run online. It doesn’t play but I think that’s because this dummy sample is too short.

Furthermore, we currently don’t support images, audio etc. in self.log(), since the api depends on the specific logger. There are efforts to standardize this #6720 So for these custom objects, you have to call self.logger.experiment.log (which is basically the same as wandb.log)

EDIT: I tried your code with DDP as well. The fix above applies.

0reactions

turiancommented, Apr 18, 2021

I see. Thank.

I’m not exactly sure how to make it more clear, but the headline “Manual Logging” is maybe a bit off-base for me. “Manual Logging to a Supported or Custom Logger”?