Intuitive space texture generation using hand tracking, speech recognition, and generative AI

Spatial Media Group, University of Aizu; Aizu-Wakamatsu, Fukushima ; Japan

^* e-mail: donabe722@gmail.com
^** e-mail: xilehence@gmail.com

Abstract

This research aims to explore new methods of intuitively redesigning room interiors using gesture, speech, and generative AI. This approach represents a new approach to interior design, allowing users to easily customize appearance of a room through voice and hand gestures. This project investigates how hand tracking, speech recognition, and generative AI can be integrated to enable intuitive and user-friendly interior texture customization in virtual spaces. Previous studies on interior design using XR have mainly used augmented reality (AR) to relocate furniture. However, in these methods, the only way to select furniture textures is to search for them in prepared furniture. Our method uses hand-tracking and speech recognition to capture a user’s desired image and employs generative AI to realize these preferences in a VR environment. The process involves scanning real-world furniture and rooms and applying AI-generated textures based on what the user communicates. The system allows users to easily visualize room interiors and modify them according to their preferences. This can enhance the traditional room design process. This method is currently restricted to texture only, but 3D model generation AI could provide additional flexibility. This method also has the potential for collaborative design work by sharing an environment.