Thank you for your great work, PLOP!

As your code, we can reproduce the performance almost the same as your paper.

Additionally, we also conduct experiments on extremely hard incremental scenarios, such as 5-1 (16 steps) and 2-1 (19 steps).

for this, we add below lines on task.py

"5-1": {
        0 : [0, 1, 2, 3, 4, 5],
        1 : [6, ],
        2 : [7, ],
        3 : [8, ],
        4 : [9, ],
        5 : [10, ],
        6 : [11, ],
        7 : [12, ],
        8 : [13, ],
        9 : [14, ],
        10: [15, ],
        11: [16, ],
        12: [17, ],
        13: [18, ],
        14: [19, ],
        15: [20, ],
    },   
    "2-1":{
        0 : [0, 1, 2],
        1 : [3, ],
        2 : [4, ],
        3 : [5, ],
        4 : [6, ],
        5 : [7, ],
        6 : [8, ],
        7 : [9, ],
        8 : [10, ],
        9 : [11, ],
        10: [12, ],
        11: [13, ],
        12: [14, ],
        13: [15, ],
        14: [16, ],
        15: [17, ],
        16: [18, ],
        17: [19, ],
        18: [20, ],
    },   

However, during the training, the loss is divergence to nan.

I already noticed that someone suffers from the loss divergence issue https://github.com/arthurdouillard/CVPR2021_PLOP/issues/8 on 15-5 task, however, I can reproduce the performance on 15-5 task in my environmental settings. Also, MiB on these extremely hard scenarios was well trained without the loss divergence, however, PLOP showed the issue.

Therefore, I wonder you also have the same issue in the extremely hard scenarios, 5-1 and 2-1. And, please tell me how can I solve that issue (e.g., which hyperparameter should be changed).

Thanks.

Hi, Thanks for your contribution. I have a problem when training PLOP on VOC dataset with setting 15-5. After successfully training the model at step 0, I trained the model at the next step. The model becomes NaN after few training iterations even at the first epoch.

Since the model M0 can be trained without any problem, I doubt that the distilling the knowledge of M0 to M1 might lead to a divergence problem for M1.

Following the papers, I used lr=0.01 for M0 and lr=0.001 for M1. Here is the setting I used. Step 0: python -m torch.distributed.launch --nproc_per_node=2 run.py --data_root /media/hieu/DATA/semantic_segmentation/PascalVOC12 --batch_size 12 --dataset voc --name PLOP --task 15-5 --overlap --step 0 --lr 0.01 --epochs 30 --method FT --pod local --pod_factor 0.01 --pod_logits --pseudo entropy --threshold 0.001 --classif_adaptive_factor --init_balanced

Step 1: python -m torch.distributed.launch --nproc_per_node=2 run.py --data_root /media/hieu/DATA/semantic_segmentation/PascalVOC12 --batch_size 12 --dataset voc --name PLOP --task 15-5 --overlap --step 1 --lr 0.001 --epochs 30 --method FT --pod local --pod_factor 0.01 --pod_logits --pseudo entropy --threshold 0.001 --classif_adaptive_factor --init_balanced.

While I am reimplementing your code with your setting given in the scripts folder, I found the results are a bit lower than the paper results(2%-5%). When I check the tensorboard for the loss, I found that from step 1, the loss is not converging and some of them are NaN.

Have you ever run into this problem?

Hi @arthurdouillard, I really enjoyed reading your work! Thanks for bringing in the domain shift aspect of CSS. I have the following doubts in the implementation of ILT, MiB and PLOP for the domain shift experiments on Cityscapes (Table 5):

1. wrt PLOP: I'm assuming the pseudo labeling will not be applicable in these experiments as the label spaces are fixed in the domain incremental scenario. So do I just use the distillation loss along with regular cross-entropy? Is my understanding correct wrt using PLOP in a domain IL scenario?
2. MiB modifies distillation and cross-entropy to tackle the background class shift issue. Since there is no such issue in the domain incremental scenario, doesn't their method get reduced to ILT (basically LwF)? I'm confused as to why there is a difference in the performance (For e.g. 59% for ILT and 61.5% for MiB in the 11-5 case).

Also, is it possible to share the joint model (traditional segmentation model) mIoU you get for Cityscapes on DeeplabV3, ResNet101? (I couldn't find this in the paper and wanted to see the drop wrt the joint one).

Hi, sorry to disturb but i have a few questions as followed.

1. Maybe you have changed some details about the experiments over Cityscapes, since it seems like MiB cannot work on the Cityscapes because there are only GT without pseudo labels. I'd appreciate it if you can tell me what the exact details you have changed over these experiments.
2. I cannot reproduce the performance that you mentioned in your paper about the Ade20k, having seen that you had reproduced the performance of your method over Cityscapes in another issue, I hope that you can reproduce it over Ade20k too and have a further discussion with me. Wish you good luck.

Describe the bug I tried to run the provided pascal VOC script using Apex optimization 01 and everything same as script except i was using a single GPU and hence changed the batch size to 24. But I got the following results

| | 1-15 | 16-20 | all | |-------------|-------|-------|-------| | Paper | 65.12 | 21.11 | 54.64 | | Code results | 58.73 | 21.6 | 49.7 |

To Reproduce start=date +%s`

START_DATE=$(date '+%Y-%m-%d')

PORT=$((9000 + RANDOM % 1000)) GPU=0 NB_GPU=1 DATA_ROOT=./data DATASET=voc TASK=15-5s NAME=PLOP METHOD=PLOP BATCH_SIZE=24 INITIAL_EPOCHS=30 EPOCHS=30 OPTIONS="--checkpoint checkpoints/step/"

RESULTSFILE=results/${START_DATE}${DATASET}${TASK}_${NAME}.csv rm -f ${RESULTSFILE}

CUDA_VISIBLE_DEVICES=${GPU} python3 -m torch.distributed.launch --master_port ${PORT} --nproc_per_node=${NB_GPU} run.py --date ${START_DATE} --data_root ${DATA_ROOT} --overlap --batch_size ${BATCH_SIZE} --dataset ${DATASET} --name ${NAME} --task ${TASK} --step 0 --lr 0.01 --epochs ${INITIAL_EPOCHS} --method ${METHOD} --opt_level O1 ${OPTIONS} for step in 1 2 3 4 5 do CUDA_VISIBLE_DEVICES=${GPU} python3 -m torch.distributed.launch --master_port ${PORT} --nproc_per_node=${NB_GPU} run.py --date ${START_DATE} --data_root ${DATA_ROOT} --overlap --batch_size ${BATCH_SIZE} --dataset ${DATASET} --name ${NAME} --task ${TASK} --step ${step} --lr 0.001 --epochs ${EPOCHS} --method ${METHOD} --opt_level O1 ${OPTIONS} done python3 average_csv.py ${RESULTSFILE}`

We cannot reproduce the performance on Cityscapes. What are your training details? We get the last mIoU is 49.73 on 11-1 task. We set the initial learning rate as 0.01 and run 60 epochs. And we set the later learning rate as 0.001 and run 60 epochs. The batch size is set to 24.

I config the same pytorch, torchvision and cuda version with you, however, I can not install apex=0.1. Can you tell you how to install it in detail? Thank you!

Describe the bug Hi , I have one question about how to predict a picture when finish the training stage, in other words, how to visualize the segmentation results in your code. I have written a simple visualization .py file, but when I load the model , I found that the model did not load successfully. If it's convenient, I would appreciate it if you could share your predict code. We look forward to your answer， thanks

To Reproduce

Dataset: voc-2012 Setting: 19-1 Command used or script used:

Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.

I'm struggling to reproduce results on CityScapes dataset like others already reported the issue. I found that in your paper you mentioned that with domain-incremental setting on CityScapes datasets you have used images from 21 cities in a particular order. However, after checking the dataset downloaded from www.cityscapes-dataset.com, i only can se 18 cities in train directory.

Now my question is as followes:

1. Where do you get images from those three extra cities, i.e. frankfurt, lindau, and munster.
2. Could you please share more details on preparing CityScapes dataset

Thanks for your time.

Hello, I have run your code model on my server. I know your code has completed the training and prediction, but I can't find the prediction picture in the file. Only the predicted score, my deep learning ability is limited, so I ask you for help on how to get the segmented image in the paper. Thank you. My email: [email protected]

Describe the bug Hi, I have run the code on 50-50-50 setting on ade20k. I want to know how to visualize on the verification set. If you have the corresponding code, can you provide it?

Thank you for your work and look forward to your reply

To Reproduce

Dataset: ADE20K Setting: 50-50-50

Describe the bug A clear and concise description of what the bug is.

To Reproduce

Dataset: City-scapes domain Setting: 1-1 task Command used or script used: I cloned the github repository and I was able to reproduce the results for FT(fine-tuning) method using the scripts. When I tried using PLOP method, after the task 0, I am getting the following error.

I am trying to reproduce 10-1 results as shown in the table below. I notice a large gap of the old class mIoU between my reproduce result (38.82) and your reported one (44.03), roughly 5 percent points. I am wondering what will cause this problem. I run the experiments with 2 x RTX 3090 GPU. I follow your original implementation except for the cuda version. I am using cuda 11.3 because cuda 10.2 does not support RTX 3090. Does it matter?

Btw, may I know what GPU model do you use? I think it requires to have at least 16G to hold a batch of 12 on each device and needs to support cuda 10.2 as well. V100? I guess.

Meanwhile, I notice a weird phenomenon that background performance drops drastically starting from the 8-th step and becomes 0 at 9-th step. I think this harms the old class performance a lot. Do you have a similar issue?

Thanks.

| step | background | aeroplane | bicycle | bird | boat | bottle | bus | car | cat | chair | cow | diningtable | dog | horse | motorbike | person | pottedplant | sheep | sofa | train | tvmonitor | 0-10 | 11-20 | all | |:----:|:----------:|:---------:|:-------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:-----------:|:------:|:------:|:---------:|:------:|:-----------:|:------:|:-----:|:-----:|:---------:|:------:|:------:|:------:| | 0 | 95.19% | 89.84% | 41.01% | 89.74% | 72.24% | 85.02% | 95.29% | 88.26% | 93.30% | 43.15% | 92.25% | x | x | x | x | x | x | x | x | x | x | 80.48% | - | 80.48% | | 1 | 92.01% | 87.25% | 35.76% | 82.88% | 67.51% | 79.44% | 94.49% | 82.55% | 88.44% | 46.25% | 90.74% | 37.38% | x | x | x | x | x | x | x | x | x | 77.03% | 37.38% | 73.73% | | 2 | 89.16% | 84.00% | 34.32% | 83.28% | 63.44% | 68.45% | 92.90% | 82.54% | 82.37% | 31.68% | 84.48% | 24.68% | 62.59% | x | x | x | x | x | x | x | x | 72.42% | 43.64% | 67.99% | | 3 | 83.97% | 79.07% | 31.92% | 73.50% | 53.19% | 51.06% | 91.18% | 80.90% | 76.82% | 17.40% | 65.26% | 18.69% | 3.93% | 22.11% | x | x | x | x | x | x | x | 64.02% | 14.91% | 53.50% | | 4 | 79.89% | 77.61% | 35.83% | 74.77% | 47.67% | 49.70% | 90.73% | 78.02% | 79.41% | 13.21% | 65.81% | 17.77% | 9.43% | 20.52% | 33.81% | x | x | x | x | x | x | 62.97% | 20.38% | 51.61% | | 5 | 84.56% | 84.34% | 37.88% | 81.72% | 53.81% | 56.75% | 90.72% | 82.48% | 82.98% | 13.77% | 69.20% | 0.57% | 0.00% | 26.10% | 58.37% | 69.71% | x | x | x | x | x | 67.11% | 30.95% | 55.81% | | 6 | 82.18% | 80.07% | 37.93% | 69.65% | 49.38% | 55.32% | 89.74% | 81.63% | 77.15% | 14.10% | 61.90% | 0.48% | 0.00% | 23.42% | 51.28% | 64.77% | 5.04% | x | x | x | x | 63.55% | 24.16% | 49.65% | | 7 | 80.42% | 77.61% | 36.88% | 50.15% | 42.00% | 53.55% | 75.10% | 80.31% | 74.55% | 10.30% | 20.94% | 0.05% | 0.00% | 24.42% | 50.90% | 57.63% | 0.00% | 19.33% | x | x | x | 54.71% | 21.76% | 41.90% | | 8 | 44.45% | 71.53% | 35.75% | 51.99% | 44.17% | 50.20% | 80.98% | 76.45% | 69.19% | 29.32% | 40.46% | 0.18% | 0.00% | 22.80% | 51.03% | 71.25% | 0.00% | 15.88% | 3.26% | x | x | 54.04% | 20.55% | 39.94% | | 9 | 0.55% | 61.80% | 35.53% | 51.15% | 44.62% | 50.52% | 69.98% | 77.75% | 57.82% | 9.92% | 16.76% | 0.05% | 0.00% | 23.56% | 46.89% | 62.95% | 0.00% | 7.37% | 1.49% | 2.27% | x | 43.31% | 16.06% | 31.05% | | 10 | 0.00% | 50.48% | 32.40% | 38.14% | 40.64% | 51.90% | 62.86% | 69.76% | 56.53% | 17.47% | 6.83% | 0.02% | 0.00% | 23.25% | 53.95% | 66.92% | 0.00% | 1.73% | 1.67% | 0.04% | 2.60% | 38.82% | 15.02% | 27.49% |

i used the script you gave and i only got 24.xx final mIoU, i don't know if it is related to specific seed?

by the way, due to the limitation of gpu memory, i used 4 gpus with 4 batch each gpu, which was equal to total batch size of 24, and the bn is iabc_sync, i thought that 4 gpus is not a issue?

any idea how i can reproduce your ade 100-10 results?

I click the link and it shows that the weight file is damaged. What is the reason? Can I upload it again? I am working on my graduation project and looking forward to your reply. Thank you! Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.