Designing the Future of Human-Computer Interaction

Abstract: Interaction with Large Language Models (LLMs) is expanding beyond strictly Conversational User Interfaces (CUIs), with systems starting to incorporate direct-manipulation mechanisms, typical of Graphical User Interfaces (GUIs), into their designs. We argue that the current landscape of human–LLM interfaces can be described as a continuum, ranging from strictly conversational to increasingly visual and interactive interfaces built on top of a basic CUI. Based on this assumption, the doctoral research presented in this paper aims to model and analyze this CUI–GUI spectrum. Grounded in a Human-Centered Design methodology and a research-through-design approach, the work investigates factors influencing users' preferences for different interface configurations along the spectrum, through comparative empirical studies, and leveraging representative interface prototypes across it.

Abstract: Agentic Web Browsers (AWBs), powered by Large Language Models (LLMs), are emerging as autonomous systems capable of navigating the Web on behalf of users. Beyond enhancing productivity, they could also offer significant promise as Assistive Technologies (ATs) for visually-impaired individuals, transforming web interaction into a fluid conversational exchange. In this paper, we present a case study with a low-vision technology expert, examining how AWBs can support visually-impaired users in web navigation. The findings show that, despite the current limitations, the navigation experience is notably fluid and flexible, underscoring the strong potential of AWBs to enhance accessibility and reduce barriers in web interaction, with implications that may extend beyond accessibility to agentic UX more broadly.

Abstract: This paper discusses voice-based direct manipulation to extend the conversational interactions with LLMs and, more generally, with intent-driven user interfaces. Inspired by recent work on visual direct manipulation in AI-assisted tools, we explore whether similar patterns can enhance Voice User Interfaces (VUIs). Such approaches can improve usability of intent formulation, especially in contexts where voice is the primary or the only interaction modality, with important implications for accessibility and social inclusion.

Abstract: This paper presents new interaction mechanisms for voice-based direct manipulation that aim to extend the conversational interaction with LLMs and, more generally, with intent-driven user interfaces. It discusses the need for these extensions and illustrates the proposed approach and prototype, and a preliminary user study.

Abstract: Recent advancements in Generative AI have contributed to the emergence of a new interaction paradigm centered on intent formulation. Despite the exceptional popularity Large Language Models (LLM) have recently gained, ensuring the user intent is accurately captured by the model remains challenging. Usability problems are even more critical when situational or permanent disabilities demand voice-based interaction. Inspired by emerging interaction techniques based on direct manipulation principles proposed for LLM GUIs, and informed by a prior user study with Blind and Visually Impaired (BVI) individuals, this thesis investigates whether direct manipulation principles can inspire the definition of new vocal interaction patterns. The thesis introduces the architecture of a prototype used to conduct validation activities with 9 participants, with findings revealing positive self-reported perceptions concerning usability and workload.