Context

Can AI help human be more creative?

Developed at the Viral Communications Group at MIT Media Lab, Neural Notes is an interactive web application that aims to help Jazz musicians become better jazz improvisers. User and the machine take turn in playing musical phrases and while doing so, help the user to become more creative. Applied to human-machine jazz trading, Neural Notes explores how real-time decision-making systems can integrate human input and machine intelligence to foster creativity.

The Problem

Jazz improvisation is deeply interactive: a call-and-response between musicians who build on one another’s ideas in real time. Yet, when practicing alone, musicians often lack that dialogue which nurtures creativity. Current practice tools exists, but they are static and predictable. The photos above show the pianist and saxophone taking turns to improvise, responding to each other dynamically, receiving feedbacks from the band.

Market Analysis

Has any existing software already addressed the challenges of practicing improvisation? Is there a demand for a new application in this area? With the user challenge in mind, I explored some existing products, learning what made them successful, and if missed opportunities presented.

User Research

Understanding the gap in existing tools, I conducted in-depth user interviews with experienced jazz musicians to identify the specific needs of a jazz improvisation application. These sessions focused on how musicians approach improvisational practice, real-time interaction, and creative flow.

In parallel, I consulted with HCI researchers at the Media Lab to understand their requirements for data collection and system logging, ensuring the platform could also serve as a research instrument for studying human–machine musical collaboration. Below are summaries of user interview sessions with the 2 groups.

Solution Ideation

“How might we create an AI partner that listens, learns, and plays with musicians — not just for them — to help them grow as improvisers?”

User Flow

I put myself into user’s shoes and ideate what would be the most intuitive user experience. Meanwhile, I made sure to communicate with project lead and users to validate if the flow make sense, ensuring changes are made at the early stage.

Initially, I had envision a "Pause" button during improvisation session, and "automatic save" the session when it ends. However, the feedback from a Jazz Musician and a HCI researcher points out the problems.

“Pause doesn’t make sense during improvisation. Everything is completed in one round during improvising. ” - Jazz Musician
“User needs to provide clear decision to “save” or “delete” a recording session for researchers to analyze clear user decision data.” - HCI Researcher

System Design

Low Fidelity Prototyping

I made sure to be empathetic to the engineers while building the structure of the application through wireframing. I prepared multiple version of the designs to discuss their feasibility, set strategic goals of the features and the designs to make sure the MVP is met first.

Ideating the record button: I first made sure it’s feasible to have the button change into different stages, then conducted A/B testing with users to pick out the best design.

Design 1

MVP Design

Though Design 1 is more informative and is designed for a stronger call to action, we decided to choose the table design as the MVP design. It’s simpler and easier to implement, but still shows the essential information — the priority is to be functional first.

Start Recording

Save Recording

Test & Iterations

While proceeding to the high-fidelity design, I realized that my initial design failed to address several core user problems through user testings, which led me to conduct two additional iterations.

High Fidelity Design

Observe Mode

The default mode for new users is Observe Mode, where users can learn how the 2 models improvise differently. They can then go on to improvise mode using the mode switcher.

Improvise Mode

Improvise Mode allows user to choose their own model setting by sliding the slider and practice improvisation with AI. When the session ends, they can re-listen to the recording and decide if they want to save it.

Recordings

Where users can view their past saved recordings, re-listen to them, and duplicate the improvise setting for a new improvisation setting.

Web Application

Call to action - go experience at the live website - to improvise page.

Neural Notes

Table of Content