Puyuan Peng

According to our database, Puyuan Peng authored at least 28 papers between 2020 and 2025.

Collaborative distances:

• Dijkstra number of four.
• Erdős number of four.

2025

TalkLess: Blending Extractive and Abstractive Speech Summarization for Editing Speech to Preserve Content and Style.

Karim Benharrak, Puyuan Peng, Amy Pavel

CoRR, July, 2025

VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation.

Puyuan Peng, Shang-Wen Li, Abdelrahman Mohamed, David Harwath

CoRR, May, 2025

ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models.

Liyan Tang, Grace Kim, Xinyu Zhao, Thom Lake, Wenxuan Ding, Fangcong Yin, Prasann Singhal, Manya Wadhwa, Zeyu Leo Liu, Zayne Sprague, Ramya Namuduri, Bodun Hu, Juan Diego Rodriguez, Puyuan Peng, Greg Durrett

CoRR, May, 2025

VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models.

Kim Sung-Bin, Jeongsoo Choi, Puyuan Peng, Joon Son Chung, Tae-Hyun Oh, David Harwath

CoRR, April, 2025

Temporally Streaming Audio-Visual Synchronization for Real-World Videos.

Jordan Voas, Wei-Cheng Tseng, Layne Berry, Xixi Hu, Puyuan Peng, James Stuedemann, David Harwath

Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2025

Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks.

Chien-yu Huang, Wei-Chih Chen, Shu-Wen Yang, Andy T. Liu, Chen-An Li, Yu-Xiang Lin, Wei-Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, William Chen, Chih-Kai Yang, Xuanjun Chen, Chi-Yuan Hsiao, Puyuan Peng, Shih-Heng Wang, Chun-Yi Kuan, Ke-Han Lu, Kai-Wei Chang, Fabian Alejandro Ritter Gutierrez, et al.

Proceedings of the Thirteenth International Conference on Learning Representations, 2025

SyllableLM: Learning Coarse Semantic Units for Speech Language Models.

Alan Baade, Puyuan Peng, David Harwath

Proceedings of the Thirteenth International Conference on Learning Representations, 2025

2024

Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks.

Chien-yu Huang, Wei-Chih Chen, Shu-Wen Yang, Andy T. Liu, Chen-An Li, Yu-Xiang Lin, Wei-Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, William Chen, Xuanjun Chen, Chi-Yuan Hsiao, Puyuan Peng, Shih-Heng Wang, Chun-Yi Kuan, Ke-Han Lu, Kai-Wei Chang, Chih-Kai Yang, Fabian Ritter Gutierrez, Ming To Chuang, Kuan-Po Huang, Siddhant Arora, You-Kuan Lin, Eunjung Yeo, Kalvin Chang, Chung-Ming Chien, Kwanghee Choi, Cheng-Hsiu Hsieh, Yi-Cheng Lin, Chee-En Yu, I-Hsiang Chiu, Heitor R. Guimarães, Jionghao Han, Tzu-Quan Lin, Tzu-Yuan Lin, Homu Chang, Ting-Wu Chang, Chun Wei Chen, Shou-Jen Chen, Yu-Hua Chen, Hsi-Chun Cheng, Kunal Dhawan, Jia-Lin Fang, Shi-Xin Fang, Kuan-Yu Fang Chiang, Chi An Fu, Hsien-Fu Hsiao, Ching Yu Hsu, Shao-Syuan Huang, Lee Chen Wei, Hsi-Che Lin, Hsuan-Hao Lin, Hsuan-Ting Lin, Jian-Ren Lin, Ting-Chun Liu, Li-Chun Lu, Tsung-Min Pai, Ankita Pasad, Shih-Yun Shan Kuan, Suwon Shon, Yuxun Tang, Yun-Shao Tsai, Jui-Chiang Wei, Tzu-Chieh Wei, Chengxi Wu, Dien-Ruei Wu, Chao-Han Huck Yang, Chieh-Chi Yang, Jia Qi Yip, Shao-Xiang Yuan, Vahid Noroozi, Zhehuai Chen, Haibin Wu, Karen Livescu, David Harwath, Shinji Watanabe, Hung-yi Lee

CoRR, 2024

VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild.

Puyuan Peng, Po-Yao Huang, Daniel Li, Abdelrahman Mohamed, David Harwath

CoRR, 2024

Neural Codec Language Models for Disentangled and Textless Voice Conversion.

Alan Baade, Puyuan Peng, David Harwath

Proceedings of the 25th Annual Conference of the International Speech Communication Association, 2024

BAT: Learning to Reason about Spatial Sounds with Large Language Models.

Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, David Harwath

Proceedings of the Forty-first International Conference on Machine Learning, 2024

SpeechCLIP+: Self-Supervised Multi-Task Representation Learning for Speech Via Clip and Speech-Image Data.

Hsuan-Fu Wang, Yi-Jen Shih, Heng-Jui Chang, Layne Berry, Puyuan Peng, Hung-Yi Lee, Hsin-Min Wang, David Harwath

Proceedings of the IEEE International Conference on Acoustics, 2024

AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models.

Yuan Tseng, Layne Berry, Yiting Chen, I-Hsiang Chiu, Hsuan-Hao Lin, Max Liu, Puyuan Peng, Yi-Jen Shih, Hung-Yu Wang, Haibin Wu, Poyao Huang, Chun-Mao Lai, Shang-Wen Li, David Harwath, Yu Tsao, Abdelrahman Mohamed, Chi-Luen Feng, Hung-Yi Lee

Proceedings of the IEEE International Conference on Acoustics, 2024

Integrating Self-Supervised Speech Model with Pseudo Word-Level Targets from Visually-Grounded Speech Model.

Hung-Chieh Fang, Nai-Xuan Ye, Yi-Jen Shih, Puyuan Peng, Hsuan-Fu Wang, Layne Berry, Hung-Yi Lee, David Harwath

Proceedings of the IEEE International Conference on Acoustics, 2024

Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos.

Changan Chen, Puyuan Peng, Ami Baid, Zihui Xue, Wei-Ning Hsu, David Harwath, Kristen Grauman

Proceedings of the Computer Vision - ECCV 2024, 2024

VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild.

Puyuan Peng, Po-Yao Huang, Shang-Wen Li, Abdelrahman Mohamed, David Harwath

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024

2023

AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models.

Yuan Tseng, Layne Berry, Yi-Ting Chen, I-Hsiang Chiu, Hsuan-Hao Lin, Max Liu, Puyuan Peng, Yi-Jen Shih, Hung-Yu Wang, Haibin Wu, Po-Yao Huang, Chun-Mao Lai, Shang-Wen Li, David Harwath, Yu Tsao, Shinji Watanabe, Abdelrahman Mohamed, Chi-Luen Feng, Hung-yi Lee

CoRR, 2023

Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Mode.

Puyuan Peng, Shang-Wen Li, Okko Räsänen, Abdelrahman Mohamed, David Harwath

CoRR, 2023

Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization.

Puyuan Peng, Brian Yan, Shinji Watanabe, David Harwath

Proceedings of the 24th Annual Conference of the International Speech Communication Association, 2023

Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model.

Puyuan Peng, Shang-Wen Li, Okko Räsänen, Abdelrahman Mohamed, David Harwath

Proceedings of the 24th Annual Conference of the International Speech Communication Association, 2023

Style-transfer based Speech and Audio-visual Scene understanding for Robot Action Sequence Acquisition from Videos.

Chiori Hori, Puyuan Peng, David Harwath, Xinyu Liu, Kei Ota, Siddarth Jain, Radu Corcodel, Devesh K. Jha, Diego Romeres, Jonathan Le Roux

Proceedings of the 24th Annual Conference of the International Speech Communication Association, 2023

Audio-Visual Neural Syntax Acquisition.

Cheng-I Jeff Lai, Freda Shi, Puyuan Peng, Yoon Kim, Kevin Gimpel, Shiyu Chang, Yung-Sung Chuang, Saurabhchand Bhati, David D. Cox, David Harwath, Yang Zhang, Karen Livescu, James R. Glass

Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, 2023

2022

Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling.

Puyuan Peng, David Harwath

CoRR, 2022

Zero-shot Video Moment Retrieval With Off-the-Shelf Models.

Anuj Diwan, Puyuan Peng, Raymond J. Mooney

Proceedings of the Transfer Learning for Natural Language Processing Workshop, 2022

Word Discovery in Visually Grounded, Self-Supervised Speech Models.

Puyuan Peng, David Harwath

Proceedings of the 23rd Annual Conference of the International Speech Communication Association, 2022

MAE-AST: Masked Autoencoding Audio Spectrogram Transformer.

Alan Baade, Puyuan Peng, David Harwath

Proceedings of the 23rd Annual Conference of the International Speech Communication Association, 2022

Fast-Slow Transformer for Visually Grounding Speech.

Puyuan Peng, David Harwath

Proceedings of the IEEE International Conference on Acoustics, 2022

2020

A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings.

Puyuan Peng, Herman Kamper, Karen Livescu

CoRR, 2020