Please use this identifier to cite or link to this item:
Title: An investigation into crossmodal and multimodal human computer interaction : handwriting, music generation and body gesture
Authors: Tang, Wai Wa
Advisors: Chan, Stephen (COMP)
Ngai, Grace (COMP)
Keywords: Human-computer interaction
Issue Date: 2015
Publisher: The Hong Kong Polytechnic University
Abstract: This thesis presents an investigation into cross-modal and multimodal human computer interaction. Human interaction is by nature multimodal and crossmodal. Human beings communicate through multiple channels of speech, gestures, facial expressions, body language, etc. In addition, cross-modal transformations, where ideas and feelings are expressed in multiple modalities, are fairly common in art. However, these modes of expression and communication have so far been mostly ignored in human computer interaction. We focus on the transformation of input modalities to alternate output modalities. The primary focus is on the transformation from writing to music, specifically Chinese calligraphy to Chinese music. We investigate both real-time as well as offline modes of transformation, and experiment with different levels of human control versus statistically driven modeling. We present evaluations that show that the music that is generated is both pleasing to the ear as well as correlates with the Chinese style of music. As an additional investigation, we also consider two other cross-modal transformations: the mapping of body motions and gestures to music manipulation operations, and the mapping of handwriting to graphics. Both transformations are motivated and demonstrated with real-life applications; preliminary evaluations show that they can encourage interaction, stimulate creativity as well as serve as a motivation to practice the input modality.
Description: x, 62 pages : color illustrations
PolyU Library Call No.: [THS] LG51 .H577M COMP 2015 Tang
Rights: All rights reserved.
Appears in Collections:Thesis

Files in This Item:
File Description SizeFormat 
991022249145703411_link.htmFor PolyU Users168 BHTMLView/Open
991022249145703411_pira_v3.pdfFor All Users (Non-printable)7.81 MBAdobe PDFView/Open
Show full item record
PIRA download icon_1.1View/Download Contents

Page view(s)

Citations as of Oct 15, 2019


Citations as of Oct 15, 2019

Google ScholarTM


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.