CG-MLLM: Captioning and Generating 3D Content via Multi-modal Large Language Models

Abstract

Large Language Models (LLMs) have revolutionized text generation and multimodal perception, but their capabilities in 3D content generation remain underexplored. Existing methods compromise by producing either low-resolution meshes or coarse structural proxies, failing to capture fine-grained geometry natively. In this paper, we propose CG-MLLM, a novel Multi-modal Large Language Model (MLLM) capable of 3D captioning and high-resolution 3D generation in a single framework. Leveraging the Mixture-of-Transformer architecture, CG-MLLM decouples disparate modeling needs, where the Token-level Autoregressive (TokenAR) Transformer handles token-level content, and the Block-level Autoregressive (BlockAR) Transformer handles block-level content. By integrating a pre-trained vision-language backbone with a specialized 3D VAE latent space, CG-MLLM facilitates long-context interactions between standard tokens and spatial blocks within a single integrated architecture. Experimental results show that CG-MLLM significantly outperforms existing MLLMs in generating high-fidelity 3D objects, effectively bringing high-resolution 3D content creation into the mainstream LLM paradigm. Beyond generation, we further observe that learning to produce 3D content transfers back to perception, strengthening the model's image-based 3D understanding.

Overview

What CG-MLLM is

A unified 3D multimodal large language model (3D MLLM) for 3D captioning and high-resolution 3D object generation.

Built upon

Qwen3-VL as the vision-language backbone and Hunyuan3D-2.1 VAE as the 3D latent tokenizer.

Architecture

Mixture-of-Transformer: a TokenAR Transformer for token-level content and a BlockAR Transformer for block-level 3D latents.

Tasks

3D content generation, 3D captioning, image-based 3D understanding, and spatial interaction across point clouds, meshes, and text.

Method Overview

Figure 1. The Pipeline of CG-MLLM. Built upon Qwen3-VL and a Hunyuan3D-2.1 VAE latent space, the multimodal architecture processes vision, text, and 3D spatial inputs to generate text and 3D spatial outputs. A TokenAR Transformer handles sequential next-token prediction, while a BlockAR Transformer performs efficient parallel block prediction, both governed by strict causal masking.

Figure 2. CG-MLLM unifies 3D understanding and 3D generation in a single 3D MLLM, supporting image understanding, point cloud understanding, mesh generation, and textual intent understanding across multiple spatial modalities.

Performance

Comparison with other methods on the image-to-3D task. For clearer visualization of geometry, materials are removed in the second row. Compared with MLLM-based methods, CG-MLLM generates more complete geometry, while achieving visual quality comparable to non-MLLM 3D generation methods.

BibTeX

@misc{huang2026cgmllmcaptioninggenerating3d,
      title={CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models},
      author={Junming Huang and Chi Wang and Letian Li and Guangkai Xu and Donglin Huang and Hao Chen and Qiang Dai and Weiwei Xu},
      year={2026},
      eprint={2601.21798},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.21798},
}