Yang Li

According to our database, Yang Li authored at least 85 papers between 2009 and 2024.

Collaborative distances:

• Dijkstra number of four.
• Erdős number of four.

2024

Generating Automatic Feedback on UI Mockups with Large Language Models.

Peitong Duan, Jeremy Warner, Yang Li, Bjoern Hartmann

CoRR, 2024

MUG: Interactive Multimodal Grounding on User Interfaces.

Tao Li, Gang Li, Jingjie Zheng, Purple Wang, Yang Li

Proceedings of the Findings of the Association for Computational Linguistics: EACL 2024, 2024

2023

Improving Workflow Integration with xPath: Design and Evaluation of a Human-AI Diagnosis System in Pathology.

Hongyan Gu, Yuan Liang, Yifan Xu, Christopher Kazu Williams, Shino Magaki, Negar Khanlou, Harry Vinters, Zesheng Chen, Shuo Ni, Chunxu Yang, Wenzhong Yan, Xinhai Robert Zhang, Yang Li, Mohammad Haeri, Xiang 'Anthony' Chen

ACM Trans. Comput. Hum. Interact., April, 2023

PLay: Parametrically Conditioned Layout Generation using Latent Diffusion.

Chin-Yi Cheng, Forrest Huang, Gang Li, Yang Li

Proceedings of the International Conference on Machine Learning, 2023

Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus.

Gang Li, Yang Li

Proceedings of the Eleventh International Conference on Learning Representations, 2023

A Zero-Shot Language Agent for Computer Control with Structured Reflection.

Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, Yang Li

Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, 2023

Enabling Conversational Interaction with Mobile UI using Large Language Models.

Bryan Wang, Gang Li, Yang Li

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023

2022

MUG: Interactive Multimodal Grounding on User Interfaces.

Tao Li, Gang Li, Jingjie Zheng, Purple Wang, Yang Li

CoRR, 2022

Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis.

Eldon Schoop, Xin Zhou, Gang Li, Zhourong Chen, Bjoern Hartmann, Yang Li

Proceedings of the CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022, 2022

Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale.

Gang Li, Gilles Baechler, Manuel Tragut, Yang Li

Proceedings of the CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022, 2022

Computational Approaches for Understanding, Generating, and Adapting User Interfaces.

Yue Jiang, Yuwen Lu, Jeffrey Nichols, Wolfgang Stuerzlinger, Chun Yu, Christof Lutteroth, Yang Li, Ranjitha Kumar, Toby Jia-Jun Li

Proceedings of the CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022, 2022

2021

VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling.

Yang Li, Gang Li, Xin Zhou, Mostafa Dehghani, Alexey A. Gritsenko

CoRR, 2021

Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models.

Forrest Huang, Gang Li, Xin Zhou, John F. Canny, Yang Li

CoRR, 2021

HelpViz: Automatic Generation of Contextual Visual MobileTutorials from Text-Based Instructions.

Mingyuan Zhong, Gang Li, Peggy Chi, Yang Li

CoRR, 2021

Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning.

Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, Yang Li

Proceedings of the UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology, 2021

HelpViz: Automatic Generation of Contextual Visual Mobile Tutorials from Text-Based Instructions.

Mingyuan Zhong, Gang Li, Peggy Chi, Yang Li

Proceedings of the UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology, 2021

Large-Scale Modeling of Mobile User Click Behaviors Using Deep Learning.

Xin Zhou, Yang Li

Proceedings of the RecSys '21: Fifteenth ACM Conference on Recommender Systems, Amsterdam, The Netherlands, 27 September 2021, 2021

Learnable Fourier Features for Multi-dimensional Spatial Positional Encoding.

Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, Samy Bengio

Proceedings of the Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, 2021

TapNet: The Design, Training, Implementation, and Applications of a Multi-Task Learning CNN for Off-Screen Mobile Input.

Michael Xuelin Huang, Yang Li, Nazneen Nazneen, Alexander Chao, Shumin Zhai

Proceedings of the CHI '21: CHI Conference on Human Factors in Computing Systems, 2021

Spacewalker: Rapid UI Design Exploration Using Lightweight Markup Enhancement and Crowd Genetic Programming.

Mingyuan Zhong, Gang Li, Yang Li

Proceedings of the CHI '21: CHI Conference on Human Factors in Computing Systems, 2021

2020

Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoders.

Yang Li, Julien Amelot, Xin Zhou, Samy Bengio, Si Si

CoRR, 2020

Multi-Stage Influence Function.

Hongge Chen, Si Si, Yang Li, Ciprian Chelba, Sanjiv Kumar, Duane S. Boning, Cho-Jui Hsieh

Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020

Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements.

Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, Zhiwei Guan

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020

Modeling Human Visual Search Performance on Realistic Webpages Using Analytical and Deep Learning Methods.

Arianna Yuan, Yang Li

Proceedings of the CHI '20: CHI Conference on Human Factors in Computing Systems, 2020

Artificial Intelligence for HCI: A Modern Approach.

Yang Li, Ranjitha Kumar, Walter S. Lasecki, Otmar Hilliges

Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, 2020

Mapping Natural Language Instructions to Mobile UI Action Sequences.

Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, Jason Baldridge

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020

2019

A Unified Framework for Data Poisoning Attack to Graph-based Semi-supervised Learning.

Xuanqing Liu, Si Si, Xiaojin Zhu, Yang Li, Cho-Jui Hsieh

CoRR, 2019

Minuet: Multimodal Interaction with an Internet of Things.

Runchang Kang, Anhong Guo, Gierad Laput, Yang Li, Xiang 'Anthony' Chen

Proceedings of the Symposium on Spatial User Interaction, 2019

A Unified Framework for Data Poisoning Attack to Graph-based Semi-supervised Learning.

Xuanqing Liu, Si Si, Jerry Zhu, Yang Li, Cho-Jui Hsieh

Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, 2019

Robustness Verification of Tree-based Models.

Hongge Chen, Huan Zhang, Si Si, Yang Li, Duane S. Boning, Cho-Jui Hsieh

Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, 2019

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks.

Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, Cho-Jui Hsieh

Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019

Area Attention.

Yang Li, Lukasz Kaiser, Samy Bengio, Si Si

Proceedings of the 36th International Conference on Machine Learning, 2019

Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks.

Patrick H. Chen, Si Si, Sanjiv Kumar, Yang Li, Cho-Jui Hsieh

Proceedings of the 7th International Conference on Learning Representations, 2019

You Look Twice: GaterNet for Dynamic Filter Selection in CNNs.

Zhourong Chen, Yang Li, Samy Bengio, Si Si

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019

Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning.

Amanda Swearngin, Yang Li

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019

2018

What is That in Your Hand?: Recognizing Grasped Objects via Forearm Electromyography Sensing.

Junjun Fan, Xiangmin Fan, Feng Tian, Yang Li, Zitao Liu, Wei Sun, Hongan Wang

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 2018

GaterNet: Dynamic Filter Selection in Convolutional Neural Network via a Dedicated Global Gating Network.

Zhourong Chen, Yang Li, Samy Bengio, Si Si

CoRR, 2018

Nonlinear Online Learning with Adaptive Nyström Approximation.

Si Si, Sanjiv Kumar, Yang Li

CoRR, 2018

GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking.

Patrick H. Chen, Si Si, Yang Li, Ciprian Chelba, Cho-Jui Hsieh

Proceedings of the Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, 2018

Time-Dependent Representation for Neural Event Sequence Prediction.

Yang Li, Nan Du, Samy Bengio

Proceedings of the 6th International Conference on Learning Representations, 2018

M3 Gesture Menu: Design and Experimental Analyses of Marking Menus for Touchscreen Mobile Interaction.

Jingjie Zheng, Xiaojun Bi, Kun Li, Yang Li, Shumin Zhai

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018

Analysis and Modeling of Grid Performance on Touchscreen Mobile Devices.

Ken Pfeuffer, Yang Li

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018

Predicting Human Performance in Vertical Menu Selection Using Deep Learning.

Yang Li, Samy Bengio, Gilles Bailly

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018

Doppio: Tracking UI Flows and Code Changes for App Development.

Pei-Yu (Peggy) Chi, Sen-Po Hu, Yang Li

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018

Investigating Cursor-based Interactions to Support Non-Visual Exploration in the Real World.

Anhong Guo, Saige McVea, Xu Wang, Patrick Clary, Ken Goldman, Yang Li, Yu Zhong, Jeffrey P. Bigham

Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, 2018

2017

Improv: An Input Framework for Improvising Cross-Device Interaction by Demonstration.

Xiang 'Anthony' Chen, Yang Li

ACM Trans. Comput. Hum. Interact., 2017

ZIPT: Zero-Integration Performance Testing of Mobile App Designs.

Biplab Deka, Zifeng Huang, Chad Franzen, Jeffrey Nichols, Yang Li, Ranjitha Kumar

Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, 2017

Rico: A Mobile App Dataset for Building Data-Driven Design Applications.

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, Ranjitha Kumar

Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, 2017

2016

Beyond the Touchscreen: An Exploration of Extending Interactions on Commodity Smartphones.

Cheng Zhang, Anhong Guo, Dingtian Zhang, Yang Li, Caleb Southern, Rosa I. Arriaga, Gregory D. Abowd

ACM Trans. Interact. Intell. Syst., 2016

Using Audio Cues to Support Motion Gesture Interaction on Mobile Devices.

Sarah Morrison-Smith, Megan Hofmann, Yang Li, Jaime Ruiz

ACM Trans. Appl. Percept., 2016

Bootstrapping User-Defined Body Tapping Recognition with Offline-Learned Probabilistic Representation.

Xiang 'Anthony' Chen, Yang Li

Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 2016

Gesture morpher: video-based retargeting of multi-touch interactions.

Ramik Sadana, Yang Li

Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services, 2016

Mogeste: mobile tool for in-situ motion gesture design.

Aman Parnami, Apurva Gupta, Gabriel Reyes, Ramik Sadana, Yang Li, Gregory D. Abowd

Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2016 ACM International Symposium on Wearable Computers, 2016

End-user development of cross-device user interfaces.

Michael Nebeling, Thomas Kubitza, Fabio Paternò, Tao Dong, Yang Li, Jeffrey Nichols

Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, 2016

Enhancing Cross-Device Interaction Scripting with Interactive Illustrations.

Pei-Yu (Peggy) Chi, Yang Li, Björn Hartmann

Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2016

The Making of Cross-Device Experiences: A Hands-on Workshop.

Tao Dong, Michael Nebeling, Daniel Afergan, Elizabeth F. Churchill, Jeffrey Nichols, Elizabeth S. Goodman, Pei-Yu (Peggy) Chi, Yang Li, Daniel Wigdor

Proceedings of the Companion Publication of the 2014 ACM Conference on Designing Interactive Systems, DIS '16, Brisbane, QLD, Australia, June 04, 2016

2015

Gesture On: Enabling Always-On Touch Gestures for Fast Mobile Access from the Device Standby Mode.

Hao Lu, Yang Li

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015

Weave: Scripting Cross-Device Wearable Interaction.

Pei-Yu (Peggy) Chi, Yang Li

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015

2014

Optimistic Programming of Touch Interaction.

Yang Li, Hao Lu, Haimo Zhang

ACM Trans. Comput. Hum. Interact., 2014

Detecting tapping motion on the side of mobile devices by probabilistically combining hand postures.

William McGrath, Yang Li

Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, 2014

Reflection: enabling event prediction as an on-device service for mobile interaction.

Yang Li

Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, 2014

HOBS: head orientation-based selection in physical spaces.

Ben Zhang, Yu-Hsiang Chen, Claire Tuna, Achal Dave, Yang Li, Edward A. Lee, Björn Hartmann

Proceedings of the 2nd ACM Symposium on Spatial User Interaction, 2014

Hierarchical route maps for efficient navigation.

Fangzhou Wang, Yang Li, Daisuke Sakamoto, Takeo Igarashi

Proceedings of the 19th International Conference on Intelligent User Interfaces, 2014

Teaching motion gestures via recognizer feedback.

Ankit Kamal, Yang Li, Edward Lank

Proceedings of the 19th International Conference on Intelligent User Interfaces, 2014

GestKeyboard: enabling gesture-based interaction on ordinary physical keyboard.

Haimo Zhang, Yang Li

Proceedings of the CHI Conference on Human Factors in Computing Systems, 2014

InkAnchor: enhancing informal ink-based note taking on touchscreen mobile phones.

Yi Ren, Yang Li, Edward Lank

Proceedings of the CHI Conference on Human Factors in Computing Systems, 2014

Gesture script: recognizing gestures and their structure using rendering scripts and interactively trained parts.

Hao Lü, James Fogarty, Yang Li

Proceedings of the CHI Conference on Human Factors in Computing Systems, 2014

Gesturemote: interacting with remote displays through touch gestures.

Hao Lü, Matei Negulescu, Yang Li

Proceedings of the International Working Conference on Advanced Visual Interfaces, 2014

2013

Open project: a lightweight framework for remote sharing of mobile applications.

Matei Negulescu, Yang Li

Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, 2013

CrowdLearner: rapidly creating mobile recognizers using crowdsourcing.

Shahriyar Amini, Yang Li

Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, 2013

Gesture studio: authoring multi-touch interactions through demonstration and declaration.

Hao Lü, Yang Li

Proceedings of the 2013 ACM SIGCHI Conference on Human Factors in Computing Systems, 2013

FFitts law: modeling finger touch with fitts' law.

Xiaojun Bi, Yang Li, Shumin Zhai

Proceedings of the 2013 ACM SIGCHI Conference on Human Factors in Computing Systems, 2013

2012

Gesture Search: Random Access to Smartphone Content.

Yang Li

IEEE Pervasive Comput., 2012

Bootstrapping personal gesture shortcuts with the wisdom of the crowd and handwriting recognition.

Tom Ouyang, Yang Li

Proceedings of the CHI Conference on Human Factors in Computing Systems, 2012

Gesture coder: a tool for programming multi-touch gestures by demonstration.

Hao Lü, Yang Li

Proceedings of the CHI Conference on Human Factors in Computing Systems, 2012

Tap, swipe, or move: attentional demands for distracted smartphone input.

Matei Negulescu, Jaime Ruiz, Yang Li, Edward Lank

Proceedings of the International Working Conference on Advanced Visual Interfaces, 2012

Gesture-based interaction: a new dimension for mobile user interfaces.

Yang Li

Proceedings of the International Working Conference on Advanced Visual Interfaces, 2012

2011

User-defined motion gestures for mobile interaction.

Jaime Ruiz, Yang Li, Edward Lank

Proceedings of the International Conference on Human Factors in Computing Systems, 2011

DoubleFlip: a motion gesture delimiter for mobile interaction.

Jaime Ruiz, Yang Li

Proceedings of the International Conference on Human Factors in Computing Systems, 2011

Gesture avatar: a technique for operating mobile user interfaces using gestures.

Hao Lü, Yang Li

Proceedings of the International Conference on Human Factors in Computing Systems, 2011

Deep shot: a framework for migrating tasks across devices using mobile phone cameras.

Tsung-Hsiang Chang, Yang Li

Proceedings of the International Conference on Human Factors in Computing Systems, 2011

Experimental analysis of touch-screen gesture designs in mobile environments.

Andrew Bragdon, Eugene Nelson, Yang Li, Ken Hinckley

Proceedings of the International Conference on Human Factors in Computing Systems, 2011

2010

Gesture search: a tool for fast mobile data access.

Yang Li

Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology, 2010

Protractor: a fast and accurate gesture recognizer.

Yang Li

Proceedings of the 28th International Conference on Human Factors in Computing Systems, 2010

2009

Beyond Pinch and Flick: Enriching Mobile Gesture Interaction.

Yang Li

Computer, 2009