CUDA out of memory when updating masks

[Dexter1066]

Hello, I get CUDA out of memory error during mask updating stages when I was training with my own dataset.

This error always occurs when first executing update_gt_mask function, in train.py, line 129, pred = net.forward(img). Before that, the training process is normal.

I am using RTX 4090 and all configurations are the same with the default ones.

Could the problem arise from my dataset and how can I avoid this problem?

Thank you!

[XinyiYing]

The error of "CUDA out of memory" exists when input images are in large size.
If the error is caused by the above reason, we suggest cutting, processing and splicing large images as a solution.

[Dexter1066]

The error of "CUDA out of memory" exists when input images are in large size. If the error is caused by the above reason, we suggest cutting, processing and splicing large images as a solution.

I notice in Update_mask dataset, no crop operation is carried. Would it be helpful to crop images and masks as TrainSetLoader does?

[XinyiYing]

You can refer to lines 57-70 of inference.py.