Timeline for Batch Normalization Layer is not learning the data semantics of a dataset comprised of datasets from different sources
Current License: CC BY-SA 4.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 31, 2023 at 15:03 | comment | added | user199590 | sure I'll try that thanks, this makes sense. | |
| Jul 30, 2023 at 14:37 | comment | added | Luca Anzalone |
Thanks. I notice that you use the same BN layer multiple times in the forward() functions. Can you try to define multiple BN layers and use each of them only once? I think the same layer to receive different statistics and may get confused during training.
|
|
| Jul 30, 2023 at 10:58 | history | edited | user199590 | CC BY-SA 4.0 |
add snippet as requiested
|
| Jul 30, 2023 at 10:57 | comment | added | user199590 | @LucaAnzalone Sure, I've edited my post to include the snippet. | |
| Jul 26, 2023 at 13:20 | comment | added | Luca Anzalone | It seems that you're doing right.. maybe there is a bug in your code. Could you post some relevant snippets? | |
| Jul 23, 2023 at 11:21 | comment | added | user199590 | Thanks @LucaAnzalone yes, I'm oversampling it such that $D_*$ is about as large as D_1 + D_2, I also tried smaller multiples. I use augmentation as well. | |
| Jul 23, 2023 at 9:54 | comment | added | Luca Anzalone | Ok, so it can be an imbalance problem since $D_*$ is much smaller. Have you tried data augmentation only on $D_*$ and/or oversampling it? | |
| Jul 22, 2023 at 17:34 | comment | added | user199590 | @LucaAnzalone thanks. I standardize just to either zero mean and standard deviation or (0.5) mean and standard deviation for all across each channel. $D_*$ is much much smaller about 200x so. | |
| Jul 22, 2023 at 13:10 | comment | added | Luca Anzalone | Is $D_*$ much smaller than $D_1 + D_2$? And also, just to check, when you standardize the data do you consider the overall stats of $D_1+D_2+D_{*train}$ or standardize each $D_i$ individually? | |
| Jul 21, 2023 at 5:49 | comment | added | user199590 | @LucaAnzalone I, see what you mean. Yes this makes sense. So, no, the training dataset is a collection of different datasets, and my test and evaluation dataset are a small subset of the small dataset I want to perform on. So if I have Datasets $D_1, D_2, D*$, I split $D*$ into train, test and eval. And I create a training set: $D_1 + D_2 + D*_{train}$ and I evaluate and test respectively on the $D*$ splits. If the parameters are fixed from the training set, they basically predict all 0's on the eval/test of D*, if in train mode, i.e. normalize to the batch of test, it works very well. | |
| Jul 20, 2023 at 13:31 | comment | added | Luca Anzalone | I understood you have a dataset made of different datasets, and that standardization is fine. What I mean is that if the overall statistics (i.e., mean and variance over all datasets) of the train are different from the test set, then BN will cause the model to perform poorly. So, if you keep updating the BN's parameters during test, the BN layers would adapt to that statistics improving performance although introducing a dependency on the test, which is unwanted. Therefore, do you split each dataset randomly to determine the test and train? | |
| Jul 19, 2023 at 18:57 | comment | added | user199590 | @LucaAnzalone yes I can, they are quite different, so I tried standardizing them, are there other better techniques? | |
| Jul 19, 2023 at 14:25 | comment | added | Luca Anzalone | Alright. So, can you check the statistics of the train and test sets? | |
| Jul 18, 2023 at 17:19 | comment | added | user199590 | Thanks for commentin. Well it's a UNet architecture, it's made by me, I've tested it on different datasets on similar tasks (segmentation) and it works. I've tried layernormalization, which works, as in it is consistent, however the performance is much worse than the batchnormalization in training mode. | |
| Jul 18, 2023 at 13:37 | comment | added | Luca Anzalone | Have you tried a different normalization layer? Also the model is a standard one (e.g. taken from literature) or is just made by you? | |
| Jul 18, 2023 at 5:35 | history | asked | user199590 | CC BY-SA 4.0 |