Why is this permutation necessary here (classification task) ?

[Joao-L-S-Almeida]

terratorch/terratorch/models/heads/classification_head.py

Line 49 in efd84db

x = x.reshape(x.shape[0], x.shape[1], -1).permute(0, 2, 1)

Any comments @CarlosGomes98 ?

[romeokienzler]

@paolo-fraccaro any idea?

[paolofraccaro]

I cannot figure out either, but also have a very vague memory that @CarlosGomes98 told me there was some factory that was needing data flipped. But cannot really remember the specifics.

[romeokienzler]

@paolo-fraccaro from your findings it seems that NHWC and NCHW are both present in TT correct? depending on the model/factory...

should we set a project wide standard? we could as we control all the factories....if yes, what would happen if we don't do it?

I've read somewhere that NCHW is the standard and better (performance wise) for nVidia

In case we don't set the standard, there is no risk in getting things wrong as the shapes won't match and we will come to know through an error, correct?

We could introduce this as a parameter (with NCHW as default) to the factory(ies)

[paolofraccaro]

Yes I think this would be helpful. Sticking to one convention would help. I am happy with NCHW (NLC for transformers). Our classification head at the moment expects NHWC/NCL instead.

[romeokienzler]

ok @paolo-fraccaro what would u suggest? change the CH or add a transform neck and have the factories add the neck to the model per default?

[paolofraccaro]

this is a way! We have the out_channels attribute from the backbones (which I believe is mandatory) therefore we should be able to permute to any default way we want to use with a neck. Either way this could be turned on/off with an attribute at factory level and clearly stated in the documentation.

[CarlosGomes98]

This code in the head strikes me as probably transformer specific... this transpose bit should be done by a neck instead. I think we can remove this line and try with a transform.

As for the convention, NCHW is the norm, but transformers have N T C (t for tokens) internally. This is needed so that internally they can efficiently perform attention (e.g. applying the same mlp to all tokens) and is why we have the necks that perform permutation if needed (see the permute neck and the neck that goes from tokens to image)