Toby Jia-Jun Li

Orcid: 0000-0001-7902-7625

Affiliations:
  • University of Notre Dame, USA


According to our database1, Toby Jia-Jun Li authored at least 101 papers between 2014 and 2026.

Collaborative distances:

Timeline

Legend:

Book 
In proceedings 
Article 
PhD thesis 
Dataset
Other 

Links

Online presence:

On csauthors.net:

Bibliography

2026
Context-aware code summary generation.
J. Syst. Softw., 2026

2025
Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification.
CoRR, August, 2025

AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multi-modal Information Between Reality and Videos.
CoRR, July, 2025

OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation.
CoRR, June, 2025

Programmer Visual Attention During Context-Aware Code Summarization.
IEEE Trans. Software Eng., May, 2025

AI-Mediated Code Comment Improvement.
CoRR, May, 2025

Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents.
CoRR, April, 2025

GLITTER: An AI-assisted Platform for Material-Grounded Asynchronous Discussion in Flipped Learning.
CoRR, April, 2025

The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections.
CoRR, April, 2025

UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agents.
CoRR, April, 2025

UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design.
CoRR, February, 2025

Uncovering Disparities in Rideshare Drivers Earning and Work Patterns: A Case Study of Chicago.
CoRR, February, 2025

'I'm Categorizing LLM as a Productivity Tool': Examining Ethics of LLM Use in HCI Research Practices.
Proc. ACM Hum. Comput. Interact., 2025

Careful About What App Promotion Ads Recommend! Detecting and Explaining Malware Promotion via App Promotion Graph.
Proceedings of the 32nd Annual Network and Distributed System Security Symposium, 2025

Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks.
Proceedings of the 30th International Conference on Intelligent User Interfaces, 2025

MetricMate: An Interactive Tool for Generating Evaluation Criteria for LLM-as-a-Judge Workflow.
Proceedings of the Joint Proceedings of the ACM IUI 2025 Workshops co-located with the 30th Annual ACM Conference on Intelligent User Interfaces (IUI 2025), 2025

Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Models.
Proceedings of the 30th International Conference on Intelligent User Interfaces, 2025

CLEAR: Towards Contextual LLM-Empowered Privacy Policy Analysis and Risk Generation for Large Language Model Applications.
Proceedings of the 30th International Conference on Intelligent User Interfaces, 2025

LADICA: A Large Shared Display Interface for Generative AI Cognitive Assistance in Co-located Team Collaboration.
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 2025

From Operation to Cognition: Automatic Modeling Cognitive Dependencies from User Demonstrations for GUI Task Automation.
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 2025

Hashtag Re-Appropriation for Audience Control on Recommendation-Driven Social Media Xiaohongshu (rednote).
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 2025

Supporting Co-Adaptive Machine Teaching through Human Concept Learning and Cognitive Theories.
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 2025

UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design.
Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2025

Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning.
Proceedings of the Findings of the Association for Computational Linguistics, 2025

Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents.
Proceedings of the Findings of the Association for Computational Linguistics, 2025

2024
Insights into Natural Language Database Query Errors: from Attention Misalignment to User Handling Strategies.
ACM Trans. Interact. Intell. Syst., December, 2024

A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code Summarization.
ACM Trans. Softw. Eng. Methodol., September, 2024

EyeTrans: Merging Human and Machine Attention for Neural Code Summarization.
Proc. ACM Softw. Eng., 2024

From Awareness to Action: Exploring End-User Empowerment Interventions for Dark Patterns in UX.
Proc. ACM Hum. Comput. Interact., 2024

Sketchar: Supporting Character Design and Illustration Prototyping Using Generative AI.
Proc. ACM Hum. Comput. Interact., 2024

ChartifyText: Automated Chart Generation from Data-Involved Texts via LLM.
CoRR, 2024

Why am I seeing this: Democratizing End User Auditing for Online Content Recommendations.
CoRR, 2024

Comparing Criteria Development Across Domain Experts, Lay Users, and Models in Large Language Model Evaluation.
CoRR, 2024

Flowy: Supporting UX Design Decisions Through AI-Driven Pattern Annotation in Multi-Screen User Flows.
CoRR, 2024

Crepe: A Mobile Screen Data Collector Using Graph Query.
CoRR, 2024

A Study on Developer Behaviors for Validating and Repairing LLM-Generated Code Using Eye Tracking and IDE Actions.
CoRR, 2024

AI Assistance for UX: A Literature Review Through Human-Centered AI.
CoRR, 2024

Developer Behaviors in Validating and Repairing LLM-Generated Code Using IDE and Eye Tracking.
Proceedings of the 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2024

SQLucid: Grounding Natural Language Database Queries with Interactive Explanations.
Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, 2024

MOCHA: Model Optimization through Collaborative Human-AI Alignment.
Proceedings of the Adjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, 2024

AutoDroid: LLM-powered Task Automation in Android.
Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, 2024

CodeGRITS: A Research Toolkit for Developer Behavior and Eye Tracking in IDE.
Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024

Challenges and Opportunities of LLM-Based Synthetic Personae and Data in HCI.
Proceedings of the Companion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing, 2024

Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation.
Proceedings of the CHI Conference on Human Factors in Computing Systems, 2024

SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers.
Proceedings of the CHI Conference on Human Factors in Computing Systems, 2024

CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models.
Proceedings of the CHI Conference on Human Factors in Computing Systems, 2024

A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration.
Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2024

An Empathy-Based Sandbox Approach to Bridge the Privacy Gap among Attitudes, Goals, Knowledge, and Behaviors.
Proceedings of the CHI Conference on Human Factors in Computing Systems, 2024

Computational Methodologies for Understanding, Automating, and Evaluating User Interfaces.
Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2024

CoCo Matrix: Taxonomy of Cognitive Contributions in Co-writing with Intelligent Agents.
Proceedings of the 16th Conference on Creativity & Cognition, 2024

MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on Videos.
Proceedings of the 16th Conference on Creativity & Cognition, 2024

2023
Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation Tasks.
CoRR, 2023

UI Layout Generation with LLMs Guided by UI Grammar.
CoRR, 2023

Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation.
CoRR, 2023

An Empathy-Based Sandbox Approach to Bridge Attitudes, Goals, Knowledge, and Behaviors in the Privacy Paradox.
CoRR, 2023

Impact of Human-AI Interaction on User Trust and Reliance in AI-Assisted Qualitative Coding.
CoRR, 2023

Empowering LLM to use Smartphone for Intelligent Task Automation.
CoRR, 2023

Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations.
CoRR, 2023

CollabCoder: A GPT-Powered Workflow for Collaborative Qualitative Analysis.
CoRR, 2023

PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data.
Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023

VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping.
Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023

Modeling Programmer Attention as Scanpath Prediction.
Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering, 2023

An Empirical Study of Model Errors and User Error Discovery and Repair Strategies in Natural Language Database Queries.
Proceedings of the 28th International Conference on Intelligent User Interfaces, 2023

SHAI 2023: Workshop on Designing for Safety in Human-AI Interactions.
Proceedings of the Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, 2023

Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations.
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

Shaping the Emerging Norms of Using Large Language Models in Social Computing Research.
Proceedings of the Computer Supported Cooperative Work and Social Computing, 2023

Designing for AI-Powered Social Computing Systems.
Proceedings of the Computer Supported Cooperative Work and Social Computing, 2023

CollabCoder: A GPT-Powered WorkFlow for Collaborative Qualitative Analysis.
Proceedings of the Computer Supported Cooperative Work and Social Computing, 2023

PaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis.
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023

The Future of Computational Approaches for Understanding and Adapting User Interfaces.
Proceedings of the Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, 2023

2022
KnowledgeShovel: An AI-in-the-Loop Document Annotation System for Scientific Knowledge Base Construction.
CoRR, 2022

AI as an Active Writer: Interaction Strategies with Generated Text in Human-AI Collaborative Fiction Writing 56-65.
Proceedings of the Joint Proceedings of the IUI 2022 Workshops: APEx-UI, 2022

A Bottom-Up End-User Intelligent Assistant Approach to Empower Gig Workers against AI Inequality.
Proceedings of the CHIWORK 2022: Symposium on Human-Computer Interaction for Work, Durham, NH, USA, June 8, 2022

StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement.
Proceedings of the CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022, 2022

Bridging the Gap Between UX Practitioners' Work Practices and AI-Enabled Design Support Tools.
Proceedings of the CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022, 2022

Computational Approaches for Understanding, Generating, and Adapting User Interfaces.
Proceedings of the CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022, 2022

It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books.
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022

Fantastic Questions and Where to Find Them: FairytaleQA - An Authentic Dataset for Narrative Comprehension.
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022

2021
It is AI's Turn to Ask Human a Question: Question and Answer Pair Generation for Children Storybooks in FairytaleQA Dataset.
CoRR, 2021

A Need-finding Study for Understanding Text Entry in SmartphoneApp Usage.
CoRR, 2021

Screen2Vec: Semantic Embedding of GUI Screens and GUI Components.
Proceedings of the CHI '21: CHI Conference on Human Factors in Computing Systems, 2021

2020
Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems.
Proc. ACM Hum. Comput. Interact., 2020

Towards Effective Human-AI Collaboration in GUI-Based Interactive Task Learning Agents.
CoRR, 2020

Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications.
Proceedings of the UIST '20: The 33rd Annual ACM Symposium on User Interface Software and Technology, 2020

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs.
Proceedings of the UIST '20: The 33rd Annual ACM Symposium on User Interface Software and Technology, 2020

Multi-Modal Interactive Task Learning from Demonstrations and Natural Language Instructions.
Proceedings of the UIST '20 Adjunct: The 33rd Annual ACM Symposium on User Interface Software and Technology, 2020

Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020

2019
PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations.
Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, 2019

End user programing of intelligent agents using demonstrations and natural language instructions.
Proceedings of the 24th International Conference on Intelligent User Interfaces: Companion, 2019

2018
Teaching Agents When They Fail: End User Development in Goal-Oriented Conversational Agents.
Proceedings of the Studies in Conversational UX Design, 2018

How End Users Express Conditionals in Programming by Demonstration for Mobile Apps.
Proceedings of the 2018 IEEE Symposium on Visual Languages and Human-Centric Computing, 2018

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions.
Proceedings of the 2018 IEEE Symposium on Visual Languages and Human-Centric Computing, 2018

Kite: Building Conversational Bots from Mobile Apps.
Proceedings of the 16th Annual International Conference on Mobile Systems, 2018

2017
PrivacyStreams: Enabling Transparency in Personal Data Processing for Mobile Apps.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 2017

End user mobile task automation using multimodal programming by demonstration.
Proceedings of the 2017 IEEE Symposium on Visual Languages and Human-Centric Computing, 2017

Programming IoT Devices by Demonstration Using Mobile Apps.
Proceedings of the End-User Development - 6th International Symposium, 2017

SUGILITE: Creating Multimodal Smartphone Automation by Demonstration.
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2017

Making End User Development More Natural.
Proceedings of the New Perspectives in End-User Development., 2017

2016
Not at Home on the Range: Peer Production and the Urban/Rural Divide.
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2016

2014
WikiBrain: Democratizing computation on Wikipedia.
Proceedings of The International Symposium on Open Collaboration, 2014

Leveraging advances in natural language processing to better understand Tobler's first law of geography.
Proceedings of the 22nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, 2014


  Loading...