Publications & Preprints

An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, Lijuan Wang
November 2023 On arXiv
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation

GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation

Preprint (arXiv 2311.07562)

Paper Demo
Raphael Schumann, Wanrong Zhu, Weixi Feng, Tsu-Jui Fu, Stefan Riezler, William Yang Wang
September 2023 On AAAI 2024
VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View

VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View

The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI 2024)

Paper Demo Code
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, Ludwig Schimdt
August 2023 On NeurIPS D&B 2023
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use

VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use

The Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS D&B 2023)

Paper Project Page Dataset Leaderboard Blog Post
Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Xuehai He, Sugato Basu, Xin Eric Wang, William Yang Wang
May 2023 On NeurIPS 2023
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

The Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023)

Paper Code Project Page
Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, William Yang Wang
May 2023 On arXiv
Multimodal Procedural Planning via Dual Text-Image Prompting

Multimodal Procedural Planning via Dual Text-Image Prompting

Preprint (arXiv 2305.01795)

Paper Code
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, Yejin Choi
April 2023 On NeurIPS D&B 2023
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text

Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text

The Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS D&B 2023)

Paper Dataset GeekWire
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, Ludwig Schmidt
March 2023 On arXiv
OpenFlamingo: An Open-Source Framework for Training Vision-Language Models with In-Context Learning

OpenFlamingo: An Open-Source Framework for Training Vision-Language Models with In-Context Learning

Preprint (arXiv 2308.01390)

Paper Blog Post (OF-v1) Blog Post (OF-v2) Code Demo
Wanrong Zhu, Xinyi Wang, Yujie Lu, Tsu-Jui Fu, Xin Eric Wang, Miguel Eckstein, William Yang Wang
February 2023 On EMNLP 2023
Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation

Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation

The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023, Short)

Paper
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, William Yang Wang
January 2023 On NeurIPS 2023
Large Language Models Are Implicitly Topic Models: Explaining and Finding Good Demonstrations for In-Context Learning

Large Language Models Are Implicitly Topic Models: Explaining and Finding Good Demonstrations for In-Context Learning

The Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023)

Paper Code
Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, William Yang Wang
October 2022 On EACL 2023
Visualize Before You Write: Imagination-Guided Open-Ended Text Generation

Visualize Before You Write: Imagination-Guided Open-Ended Text Generation

The 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2023, Findings)

Paper Code
An Yan, Jiacheng Li, Wanrong Zhu, Yujie Lu, William Yang Wang, Julian McAuley
October 2022 On arXiv
CLIP also Understands Text: Prompting CLIP for Phrase Understanding

CLIP also Understands Text: Prompting CLIP for Phrase Understanding

Preprint (arXiv 2210.05836)

Paper
Yujie Lu, Weixi Feng, Wanrong Zhu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, William Yang Wang
June 2022 On ICLR 2023
Neuro-Symbolic Causal Language Planning with Commonsense Prompting

Neuro-Symbolic Causal Language Planning with Commonsense Prompting

The 11th International Conference on Learning Representations (ICLR 2023, Spotlight)

Paper Project Code
Wanrong Zhu, Bo Pang, Ashish V. Thapliyal, William Yang Wang, Radu Soricut
April 2022 On COLING 2022
End-to-end Dense Video Captioning as Sequence Generation

End-to-end Dense Video Captioning as Sequence Generation

The 29th International Conference on Computational Linguistics (COLING 2022)

Paper Video Poster
Yujie Lu, Wanrong Zhu, Xin Eric Wang, Miguel Eckstein, William Yang Wang
April 2022 On NAACL 2022
Imagination-Augmented Natural Language Understanding

Imagination-Augmented Natural Language Understanding

The 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2022, Oral)

Paper Code
Wanrong Zhu, Xin Eric Wang, An Yan, Miguel Eckstein, William Yang Wang
June 2021 On EACL 2023
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation

ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation

The 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2023, Findings)

Paper
Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Eric Wang, Qi Wu, Miguel Eckstein, William Yang Wang
March 2021 On NAACL 2022
Diagnosing Vision-and-Language Navigation: What Really Matters

Diagnosing Vision-and-Language Navigation: What Really Matters

The 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2022, Oral)

Paper Code Video Slides
Wanrong Zhu, Xin Eric Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang
January 2021 On EACL 2021
Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation

Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation

The 16th conference of the European Chapter of the Association for Computational Linguistics (EACL 2021)

Paper Code Video & Slides Poster
Wanrong Zhu, Xin Eric Wang, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang
October 2020 On EMNLP 2020
Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations

Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations

The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020, Short)

Paper Video & Slides
Zhiting Hu, Haoran Shi, Bowen Tan, Wentao Wang, Zichao Yang, Tiancheng Zhao, Junxian He, Lianhui Qin, Di Wang, Xuezhe Ma, Zhengzhong Liu, Xiaodan Liang, Wanrong Zhu, Devendra Singh Sachan, Eric P. Xing
September 2018 On ACL 2019 System Demonstration
Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation

Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation

The 57th Annual Meeting of the Association for Computational Linguistics:System Demonstrations (ACL 2019 System Demonstration)

Paper Code Docs
Wanrong Zhu, Zhiting Hu, Eric P. Xing
January 2015 On arXiv
Text Infilling

Text Infilling

Preprint (arXiv 1901.00158)

Paper Code