'assert (boxes1[:, 2:] >= boxes1[:, :2]).all()' happened when training

See original GitHub issue

Thanks for amazing work I have questions when training with your code, assert (boxes1[:, 2:] >= boxes1[:, :2]).all() happened in function generalized_box_iou After reading the code i find that boxes1 is the predictd bbox from a MLP layer, which i think the above assertion may happen during early training time, and then break the training. I wonder if there are Mechanism that can make sure to avoid this happen

My Environment:

Provide your environment information using the following command: Collecting environment information… PyTorch version: 1.4.0 Is debug build: No CUDA used to build PyTorch: 10.0

OS: Ubuntu 16.04.5 LTS GCC version: (Ubuntu 4.9.3-13ubuntu2) 4.9.3 CMake version: version 3.16.2

Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 10.0.130 GPU models and configuration: GPU 0: TITAN Xp GPU 1: TITAN Xp GPU 2: TITAN Xp GPU 3: TITAN Xp

Nvidia driver version: 410.48 cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries: [pip3] numpy==1.17.1 [pip3] torch==1.4.0 [pip3] torchfile==0.1.0 [pip3] torchvision==0.5.0 [conda] mkl 2019.4 243 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main [conda] pytorch 1.4.0 py3.6_cuda10.0.130_cudnn7.6.3_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch [conda] torch 1.0.0 <pip> [conda] torchfile 0.1.0 <pip> [conda] torchvision 0.5.0 py36_cu100 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch

Issue Analytics

  • State:open
  • Created 3 years ago
  • Reactions:7
  • Comments:23 (2 by maintainers)

github_iconTop GitHub Comments

4reactions
ravivcommented, Jun 24, 2020

@LovPe I was getting this error when the learning rate was too high.

2reactions
eslambakrcommented, Sep 28, 2021

I am sharing my experience may be it will be beneficial for you. I got this error when the num_classes is set in a wrong way. When I fixed it the error was resolved.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Source code for transformers.models.detr.modeling_detr
It is a list of dictionaries containing the two above keys (:obj:`logits` and ... assert (boxes2[:, 2:] >= boxes2[:, :2]).all() iou, union =...
Read more >
DETR项目--github--facebookresearch/detr_zzfive的博客
All Rights Reserved """ DETR Transformer class. ... The process happens in two steps: 1) we compute hungarian assignment between ground ...
Read more >
CUDA Error: Device-Side Assert Triggered: Solved | Built In
How to Fix a CUDA Error: Device-Side Assert Triggered in PyTorch ... The following two reasons cause a CUDA error to occur:.
Read more >
mrcnn/utils.py · hellotrick/Mask_RCNN - Gitee.com
"""Computes IoU overlaps between two sets of boxes. boxes1, boxes2: [N, (y1, x1, y2, x2)]. For ...
Read more >
Data — MONAI 1.0.1 Documentation
During training call set_data() to update input data and recompute cache content. ... default to 2 . this arg is used by torch.save...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found