Inference inputs multiple modalities other than text at once

[xxrbudong]

Hello, I would like to ask, the current code seems to support only one modality and text modality at a time of inference, is it possible to input multiple modal data (such as audio, video and text) at a time of inference?

[csuhan]

The current model is not trained on joint multimodal data, so it may not perform well at the test time.

[Cece1031]

The current model is not trained on joint multimodal data, so it may not perform well at the test time.
But I see you run the test on Music-AVQA in thesis, could u tell me how you manage to use three modalities to generate answers?Thank u very much!

[csuhan]

Hi @Cece1031 , hope the script in #29 can help you.