ERROR - The testing results of the whole dataset is empty.

See original GitHub issue

Describe the bug When I run faster_rcnn_r101_fpn_1x_coco, I can get the bbox evaluation, but I run faster_rcnn_hrnetv2p_w40_1x_coco, faster_rcnn_x101_64x4d_fpn_1x_coco I got the error. ERROR - The testing results of the whole dataset is empty. Use tools/test.py to get the same error

Environment

python tools/train.py configs/faster_rcnn/faster_rcnn_r101_fpn_1x_coco.py is OK.
python tools/train.py configs/faster_rcnn/faster_rcnn_x101_64x4d_fpn_1x_coco.py is ERROR.
python tools/train.py configs/hrnet/faster_rcnn_hrnetv2p_w40_1x_coco.py is ERROR. I use Tesla V100-PCIE-32GB, The three models are trained simultaneously！

I need help！

Issue Analytics

State:
Created 3 years ago
Comments:9

Top GitHub Comments

4reactions

hongrui16commented, Jul 26, 2020

I got the same error, when the epoch is smaller than 4, the evaluation can be implemented. After the epoch increased to 5, the loss wan nan and got error, ERROR - The testing results of the whole dataset is empty, during evaluation.

0reactions

dream-in-nightcommented, Aug 11, 2020

i have meet the same problem, i slove it by create another conda enviroment. the problem come when i use my 1060 gpu, and if i run the same code in colab, the problem disappared .