Inquiry about the BEiT Model Implementation

[AneesaI]

First of all, thank you for the great efforts that you put into creating the keras_cv_attention_models repository.
I am writing to inquire about a specific component within the BEiT model's architecture, particularly including the ChannelAffine layer within the attention_mlp_block. I am interested in understanding whether this layer is part of the original BEiT architecture or represents an additional layer introduced in your implementation.

Also, according to BEiT architecture , the model includes "masked image modeling head" but I couldn't find it in your implementation, could you please elaborate more about your Beit implementation.

Note that an email sent to you in this regard from [email protected]

Thanks,

[leondgarse]

• The ChannelAffine layers are corresponding to the gamma values in modeling_finetune.py#L179-L180, which are commonly used in many models. It's just a weight matching the last dimension of its inputs.
• An update here adding use_masked_positions for beit added for the masked model, not sure if it's what you want. Note an additional mask_inputs is added, also affecting model output block.

  from keras_cv_attention_models import beit
  mm = beit.BeitV2BasePatch16(use_mask_inputs=True)
  print(mm.input_shape, mm.output_shape)
  # [(None, 224, 224, 3), (None, 196)] (None, 1000)
  
  print(mm([tf.ones([1, 224, 224, 3]), tf.random.uniform([1, 196], 0, 2, dtype='int64')]).shape)
  # (101, 1000)

  May also notice there the mask_token also using the ChannelAffine layer, which is related with modeling_pretrain.py#L42.

  aa = mm.get_layer('mask_token')
  print(aa.weights[0].shape)
  # (768,)