Gus XIA

My Research: Towards a more creative, expressive, interactive world via Music AI.

My research journey so far has unfolded in three stages. The first stage centered on Music AI technologies, including music information retrieval, expressive performance modeling, music robotics, and haptic guidance for flute education. The second stage turned toward interpretability and control in Music AI, with a focus on self-supervised learning, representation disentanglement, style transfer, hierarchical modeling, and symbolic music generation.

In the third, current stage, music serves as a lens for uncovering the fundamental organizing principles of human intelligence. My team and I are developing two related ideas: Function Alignment, which studies how System 1 and System 2 processes work together, and Emergent Language, which asks how symbolic systems can arise from intuitive processes. I believe these questions are central to understanding aesthetics, creativity, and human expression.

For me, music is not only an application domain for AI, but also a privileged window into the mechanisms through which humans make meaning, create beauty, and build culture.

About me: A computer scientist and professional musician.

I am an Associate Professor in Machine Learning at MBZUAI, where I direct the Music X Lab. I received my Ph.D. from the Machine Learning Department at Carnegie Mellon University, advised by Prof. Roger Dannenberg, and was a Neukom Fellow at Dartmouth from 2016 to 2017. I received my undergraduate degree in Information Science, with a minor in Psychology, from Peking University.

I am also a professional DI and XIAO player, keeping music at the center of my life. At Peking University, I was the principal soloist of the Chinese Music Institute (CMI), and served as its president and assistant conductor. I held my solo concert in 2010 and a Music AI concert in 2022. (See Music Events)


Notice:
Summer 2022: Music X Lab moved from NYU Shanghai to MBZUAI at Abu Dhabi. I still hold a courtesy appointment at NYU Global.

I am always looking for passionate, self-motivated students to join Music X Lab.

Representative studies organized by topics  (See here or Google Scholar for a more comprehensive publication list).

Big Picture & Highlights

Position paper on a new theory of mind:

Latest studies on Function Alignment and Emergent Language:

[Intro to Music X Lab's research (2022)]

Deep Music Generation (with interpretability and control)

Overview:

[Blog, Analogy vs. Prediction: a Debate on the Philosophy of Music Automation], [Music Style Transfer: A Position Paper]

Representation learning and disentanglement for symbolic music, from monophonic to polyphonic:

[EC2-VAE], [PianoTree VAE], [Poly-dis]

Multimodal music representation learning, bridging analysis and creation:

[Chimera: A Unified Model for Separation, Transcription and Synthesis], [A2S: Audio-to-symbolic Arrangement],
[BUTTER], [Cinematic GPT-4]

Accompaniment arrangement via music representation learning:

LLMs (for Music)

Overview:

[Seminar Discussion, Interpretability in LLMs and "Functional Alignment": video and note]

Foundation models and benchmarks:

[MERT], [MARBLE], [MuPT], [YuE]

Controlling and fine-tuning music LLMs:

[Loop Copilot], [Coco-Mulla], [Air-Gen], [ViTex]

Interpretability, copyright, culture, and human perspective:

[Do LLMs "Feel"?], [Symbolic Concept Probing],
[Human-centered LLM-agent UI], [Deep Recombinant Transformer], [Who Gets Heard?], [Music for All]

Hierarchical Representation Learning

Overview:

[Blog, Key Issues in Hierarchical Modeling: coming soon]

Long-term music modeling via compositional hierarchy:

[Hierarchical-EC2-VAE], [Whole-song Hierarchical Generation]

Modeling context-dependency: content, form, and motif development:

[Transformer VAE], [Content-Form Disentanglement], [Sequence Attention], [Motif Learning], [TOMI]

Human-AI Music Co-creation

Overview:

[Blog, The Many Ways to Control a Generative Model: coming soon]

Inpainting and refinement, note-by-note, chunk-by-chunk, and factor-by-factor:

[MuseBERT], [Phrase Inpainting], [Mixed-level Inpainting]

Unifying internal and external control via diffusion models:

[Polyffusion], [Whole-song Gen by Cascaded Diffusion]

Computer-aided Music Learning via Multimodal feedbacks
D. Chin, Y. and G. Xia A Computer-aided Multimodal Music Learning System with Curriculum: A Pilot Study
[PDF]
2021 Proc. 22nd The International Conference on New Interfaces for Musical Expression (NIME)
D. Chin, Y. Zhang, and G. Xia Hyper-hybrid Flute: Simulating and Augmenting How Breath Affects Octave and Microtone
[PDF]
2021 Proc. 21st The International Conference on New Interfaces for Musical Expression (NIME)
Y. Li, Z. Piao, and G. Xia A Wearable Haptic Interface for Breath Guidance in Vocal Training
[PDF]
2021 Proc. 21st The International Conference on New Interfaces for Musical Expression (NIME)
D. Chin, Y. Zhang, T. Zhang, J. Zhao, and G. Xia Rainbow Score: A Visual-centered Multimodal Flute Tutoring System
[PDF]
2020 Proc. 20th The International Conference on New Interfaces for Musical Expression (NIME)
Y. Zhang, Y. Li, D. Chin, and G. Xia Adaptive Multimodal Music Learning via Interactive-haptic Instrument
[PDF]
2019 Proc. 19th The International Conference on New Interfaces for Musical Expression (NIME)
G. Xia, C. Jacobsen, Q. Chen, X-D. Yang, and R. Dannenberg ShIFT: A Semi-haptic Interface for Flute Tutoring
[PDF]
2018 Proc. 18th The International Conference on New Interfaces for Musical Expression (NIME)
Expressive and Interactive Performance
M. Xu, Z. Wang, and G. Xia Modeling Perceptual Loudness of Piano Tone:Theory and Pratice
[PDF coming soon]
2022 Proc. 23th the International Conference on Music Information Retrieval (ISMIR)
M. Xu, Z. Wang, and G. Xia Transferring Piano Performance Control Across Environments
[PDF] [Poster]
2019 Proc. 20th the International Conference on Music Information Retrieval (ISMIR)
G. Xia, and R. Dannenberg Duet Interaction: Learning Improvisation Techniques for Automatic Accompaniment
[PDF]
2017 Proc. 17th The International Conference on New Interfaces for Musical Expression (NIME)
S. Dai, and G. Xia, Computational Models for Common Pipa Techniques [PDF coming soon] 2017 Proc. 5th  National Conference on Sound and Music Technology (CSMT) Best student paper award
G. Xia, K. Mao, M. Fu, S. Cosentino, G.Trovato, S. Sessa, A. Takanishi, and R. Dannenberg Expressive Humanoid Robot for Automatic Accompaniment.
[PDF] [Slides]
2016 Proc. 13th the International Conference on Sound and Music Computing (SMC)
G. Xia, Y. Wang, R. Dannenberg, and G. Gordon Spectral Learning for Expressive Interactive Ensemble Music Performance.
[PDF] [Slides] [Audio Demo]
2015 Proc. 16th the International Conference on Music Information Retrieval (ISMIR)
G. Xia & R. Dannenberg Duet Interaction: Learning Musicianship for Automatic Accompaniment.
[PDF] [Slides] [Audio Demo] [Data]
2015 Proc. 15th the International Conference on New Interfaces for Musical Expression (NIME)
M. Fu, G. Xia, R. Dannenberg, and L. Wasserman A Statistical View on the Expressive Timing of Piano Rolled Chords.
[PDF] [Poster]
2015 Proc. 16th the International Conference on Music Information Retrieval (ISMIR)
G. Xia, J. Tay, R. Dannenberg, and M. Veloso Autonomous Robot Dancing Driven by Beats and Emotions of Music.
[PDF] [Slides] [Demo1] [Demo2]
2012 Proc. 12th International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS)
D. Liang, G. Xia, and R. Dannenberg A Framework for Coordination and Synchronization of Media.
[PDF] [Slides] [Talk + Demo]
2011 Proc. 11th the International Conference on New Interfaces for Musical Expression (NIME)
Music Information Retrieval
X. Liu, D. Chin, Y. Huang, and G. Xia Learning Interpretable Low-dimensional Representation via Physical SymmetryMeasures
[PDF]
2023 Proc. 37th Conference on Neural Information Processing Systems (NeurIPS)
J. Jiang, D. Chin, Y. Zhang, and G. Xia Learning Hierarchical Metrical Structure Beyond Measures
[PDF]
2022 Proc. 23nd the International Conference on Music Information Retrieval (ISMIR)
L. Lin, Q. Kong, J. Jiang, and G. Xia A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis.
[PDF]
2021 Proc. 22nd the International Conference on Music Information Retrieval (ISMIR)
J. Jiang, K. Chen, W. Li, and G. Xia Large Vocabulary Chord Transcription via Chord Structure Decomposition.
[PDF]
2019 Proc. 20th the International Conference on Music Information Retrieval (ISMIR)
G. Xia, T. Huang, Y. Ma, R. Dannenberg, and C. Faloutsos MidiFind: Similarity Search and Popularity Mining in Large MIDI Databases.
[PDF] [Website: online MIDI search]
2014 Sound, Music, and Motion, LNCS 2014, pp 259 - 276
G. Xia, D. Liang, R. Dannenberg, and M. Harvilla Segmentation, Clustering, and Display in a Personal Audio Database for Musicians.
[PDF] [Poster]
2011 Proc. 12th the International Conference on Music Information Retrieval (ISMIR)
Selected music events
Title Description Year Institution
Music AI concert A Music AI concert [Video]. Summer 2022 MBZUAI and Music X Lab
Sally Garden Human-Robot Interactive Performances in collaboration with the world-leading saxphone robot [Video] and piano robot [Video]. Fall 2015 Carnegie Mellon, Waseda University, and TeoTronica Inc.
Laptop Orchestra
A collection of distributed performers who play laptop computers as instrument. [Webpage] [Poster] Summer 2012 Carnegie Mellon
DI & XIAO Solo Concert
Some representitive pieces of DI and XIAO were performed, including: [Video1] [Video2] [Video3] [Video4] [Video5] Summer 2010 Peking University

 

 

Webpage co-created with ChatGPT/Codex, last modified on June 17, 2026.