Video-ChatGPT
- Paper: Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
- GitHub Link
- Publisher:
Arxiv
- Author Affiliation:
Mohamed bin Zayed University of AI
- Functional Division
- Understanding
- Generation
- Design Division
- Tool-using
- End-to-end
- Input Modalities $\rightarrow$ Output Modalities
(I: Image, V: Video, A: Audio, 3D: Point Cloud, T: Text, ID: Document understanding, IB: Output bounding box, IM: Output segmentation mask, IR: Output retrieved images)- V+T $\rightarrow$ T
- Model Architecture
(Input $\rightarrow$ Modality Encoder $\rightarrow$ Input Projector $\rightarrow$ LLM Backbone $\rightarrow$ Output Projector $\rightarrow$ Modality Generator $\rightarrow$ Output)- Modality Encoder
I: CLIP ViT-L/14
- Input Projector
Linear Projector
- LLM Backbone
Vicuna-v1.1
- Output Projector
None
- Modality Generator
None
- Modality Encoder
- Datasets Scale
- Pre-training Stage
Not report
- Instruction-tuning Stage
Not report
- Pre-training Stage
This post is licensed under CC BY 4.0 by the author.