A fuzzy gestural AI for expressive interactivity in multi-touch DMIs based on Laban’s descriptors

Abstract

One of the main research areas in the field of musical human–AI interactivity is how to incorporate expressiveness into interactive digital musical instruments (DMIs). In this study we analyzed gestures rooted in expressiveness by using AI techniques that can enhance the mapping stage of multitouch DMIs. This approach not only considers the geometric information of various gestures but also incorporates expressiveness, which is a crucial element of musicality. Our focus is specifically on multitouch DMIs, and we use expressive descriptors and a fuzzy logic model to mathematically analyze performers’ finger movements. By incorporating commonly used features from the literature and adapting some of Rudolf Laban’s descriptors—originally intended for full-body analysis—to finger-based multitouch systems, we aim to enrich the mapping process. To achieve this, we developed an AI algorithm based on a fuzzy control system that takes these descriptors as inputs and maps them to synthesis variables. This tool empowers DMI designers to define their own mapping rules based on expressive gestural descriptions, using musical metaphors in a simple and intuitive way. Through a user evaluation, we demonstrate the effectiveness of our approach in capturing and representing gestural expressiveness in the case of multitouch DMIs.