- Roger Dannenberg, co-founder of audio editing software Audacity, and Jorge Sastre, director of the collaborative Soundcool project, are experts in the intersection between music, science, and technology.
- Soundcool plays a pivotal role in facilitating collaborative music creation, enhancing educational experiences, and revolutionising stimulation treatment for neurodegenerative diseases.
- In this interview with Research Features, the pair discuss the development of this collaboration, how technology and their field are changing, and their future plans.
In this exciting interview with Research Features, we delve into the intersecting worlds of technology and music with Roger Dannenberg, co-founder of Audacity, and Jorge Sastre, collaborator and director of the Soundcool project. Dannenberg and Sastre discuss the educational impact of Soundcool, highlighting its ability to foster creativity and collaboration among students, and explore the potential of technology in reshaping music education and even extending into healthcare realms like neurodegenerative disease therapy.
Roger, you’ve had a diverse and fascinating career with significant contributions to computer science and music, and are a Professor of Computer Science at Carnegie Mellon University, Pittsburgh, USA. Could you tell us a bit about Audacity and how this has impacted your role as a professor?
Audacity is an easy-to-use, open-source audio editor to record and edit music, convert audio files from one format to another, create audiobooks and blogs, and much more. Audacity was born out of a simple need in my lab, but its impact on democratising music production has been immense.
My student at the time, Dominic Mazzoni, thought it would be a small additional effort to add some cut/copy/paste commands to the display program, and Audacity was born. Audacity was largely developed by a team, and Dominic Mazzoni provided most of the leadership over many years, so aside from occasional contributions, I’ve been able to enjoy using Audacity in my research while others have done the heavy lifting. The recognition from being a founder of the project is great and helps attract good students, but most people have no idea where Audacity is from.
How has the intersection of computer science and music fields changed music creation and consumption, and what inspired you to bridge these disciplines?
The changes in these fields have been enormous, especially if you go back to before fast personal computers and fast networks emerged about 25 years ago. Professional audio recording and production required expensive equipment and expert engineers. It’s exciting to me that so many creative young people have access to professional tools, and often, Audacity is a first introduction to this amazing world.
Audacity was born out of a simple need in my lab, but its impact on democratising music production has been immense.
As a computer scientist, what really got me started and still excites me is not to make traditional recording practice better and cheaper, but to find new applications for computation in music, such as interactive compositions where the computer composes in collaboration with a live performer, or creating new sounds that do not exist in acoustic instruments. I am also a musician and composer, so bringing computers into my music practice always seemed natural.
What innovative approaches or technologies do you find particularly promising for nurturing the next generation of musicians?
First, a revolution has already occurred where control over access to music has been taken from recording companies whose entire business model hinged on a limited inventory of physical recordings. Now, everyone can find recordings and performances from all styles, eras, and cultures for free on the internet. Combined with the music production capabilities of modern software and computers, the capacity for artistic freedom and independence is unprecedented.
Audacity was recently acquired by Muse Group, Inc, and while I cannot speak for the company, the general direction for the future is to grant music creators licensed access to music content, with free and open tools like Audacity and pre-negotiated low costs. Students can access guitar chords and fingerings, community bands and choirs can make their own arrangements of popular songs, DJs can remix commercial audio tracks. Who knows what creative works will emerge!
The elephant in the room is artificial intelligence (AI), which promises or threatens many possibilities, depending on who you talk to. For the most part, AI has failed to produce anything of professional quality except in the hands of real artists, who have always been able to use new technologies, whether it’s a camel-hair paintbrush, a piano, a tape recorder, or even AI. A big focus in AI now is ‘controllability’, which is more or less a clear admission that current AI, at least alone, is unable to set and achieve imaginative goals or even develop a broad understanding of a complex task. Nevertheless, AI is already performing some specific tasks in music and audio production, and can play a role in less demanding applications like creating background music for amateur videos or giving basic instructional feedback to music student beginners.
Your recent paper Mobile devices and sensors for an educational multimedia opera project discusses Soundcool. What inspired this research collaboration, and how can tools like Soundcool reshape our understanding of teaching and learning processes?
Roger: Serendipity shapes a lot of research and collaborations. I met Jorge through Leonardo Balada in 2012, another professor at Carnegie Mellon, and because of our common interest in computer music, we have been working together since. The opera was an opportunity to use Soundcool in a large project, and when I was visiting Valencia, Jorge proposed that we work on it together. How could I say no? It has been exciting work. Soundcool is very simple, open-ended and flexible. It especially encourages young people to be creative with sound. While there is a lot of computer-based instruction that tends toward question-and-answer sorts of interaction, Soundcool points the way to a focus on creativity and collaboration. I hope it inspires educators and children alike.
Jorge: I went for a scholar visit with Roger in 2013, with the proposal of creating a modular system for collaborative creation with smartphones and tablets. When the first version was released, we called it Soundcool. In 2016, we invited Roger to Spain to see classes with Soundcool and he was so impressed that he wanted to collaborate more actively. At that time, the Soundcool team was working on an opera project. Its objective was to make the students part of an opera creation and performance using the Soundcool system, which would allow students to collaborate and make the most of their creativity for a beautiful performance.
The story was based on a traditional tale from the Valencian Community written by Enric Valor. There is a similar popular tale in Italy, called Il Drago dalle sette teste (The Seven-Headed Dragon). Magical elements and beings appear in the story, such as a seven-headed dragon, the Mother of the Fishes, a witch, spells, etc. There were many sounds to ideate and create. It was a stimulating and motivating activity, since the sounds the students created would appear in a theatrical opera performance. Moreover, as members of the orchestra, the students would trigger the sounds and control parameters of real-time effects with Soundcool using tablets.
Soundcool was applied in a multidisciplinary international project involving countries such as Spain, Romania, Mexico, and the USA. What were the key takeaways and lessons learned from these international collaborations, and how can these experiences inform future projects?
The objective of the project was to get students involved, creative, and responsible, enabled by the Soundcool system. Soundcool is like ‘LEGO’ for sound and image live creation. It has very simple ‘pieces’ (modules) that work in a computer and can be interconnected. They allow to build from simple to complex live audio and/or video performances. These modules can be controlled with smartphones and tablets, technologies with which younger generations are very familiar, and their use enables collaborative creation.
Students produced their own effects, scenography, and costume. In Spain for example, there were performances using recycled materials artistically, in support of the Sustainable Development Goals.
We think that one of the main reasons for the success of this project is giving freedom to the students and the production teams to create their own version. I think this must be a key point in all future educational projects, but also in other kinds of projects: to let participants contribute collaboratively to the creation and final performances, motivating students through ownership and opportunities to express themselves.
The Opera La Mare dels Peixos, created with the help of Soundcool, is highlighted in your paper as a significant project that extended traditional orchestral instruments and opera singing with electronics. Can you delve into the creative and educational impact of this project and how it influenced the roles of students in sound and visual creation within the opera?
One of the interesting ideas in the opera is that it is, in some respects, unfinished. There are intentional ‘holes’ in the composition to be filled with electronic sounds designed by young people. In this way, each opera production invites participation by the community, opening a doorway to the opera experience. Of course, every opera production has a director, set and costume designs and other artistic contributions. Still, productions often appear as complete works handed down from the artistic stratosphere, leaving the opera goers with no sense of participation or ownership. We love to see great artistic work, but we also feel the classical world needs to cultivate a stronger connection to the community as well as the next generation of artists.
What are the potential future applications of Soundcool and similar technologies, and how might they continue to evolve to address various domains, including healthcare?
Soundcool is available for free and can be downloaded and used all over the world.
Soundcool continues to be used for education. In 2015, we started to use it for functional diversity (such as autism and Down syndrome) and a PhD thesis dealing with this subject was presented. The results of motivation, and involvement in this case were similar to those in education. Then we thought this result could be extended to stimulate people with Alzheimer’s and other neurodegenerative diseases, and started to test Soundcool for neurodegenerative diseases therapy. Since Soundcool was adapted to work remotely in the COVID-19 pandemic, we could then show the system to neurologists and Alzheimer’s associations, connecting neurologists and therapists to the system by just downloading the app.
Soundcool points the way to a focus on creativity and collaboration in education, inspiring both educators and children alike.
One of the first collaborative activities with them was recording the phrases and sounds of a Mascletà, consisting of sounds of fireworks, cracks, and whistles. The sound is most important since it is performed in the day: it is like a percussion musical piece with different sections.
The first time Roger came to visit us, he was able to witness the participants’ emotion and excitement firsthand. Then, throughout the pandemic, the Alzheimer Association AFA Canals (Spain) liked the idea of preparing a virtual one which evolved into recording the sounds, drawing the pictures for each firework element, putting them together in a graphical score, and finally performing the Mascletà with Soundcool in tablets.
We also have held several concerts, first in October 2021, for people with Alzheimer’s, facilitating them to contribute from their Alzheimer’s associations with video creations, including parameters such as brightness or hue for videos, video blendings, video switching, for the music in a theatre, while they watched the concert and video creations live through a videoconferencing software. We have since performed several times in Spain with users from AFA Canals and The Alzheimer Association of Valencia (AFAV), as well as Mexico City.
Are there any upcoming projects that you are both particularly excited about?
From April to July 2024, we will perform a new concert with people with Alzheimer’s and other neurodegenerative diseases or with healthy people for prevention. It will be in a scholar visit to the New York University, supported by the Fulbright Commission and the Spanish Ministry Salvador de Madariaga grant PRX22-00393. Through this project, we want to connect users from Spain and the USA in New York. Soundcool has been used also for ADHD and we think about the possibility to extend its use for the stimulation of neurocognitive diseases in general: In the previous Alzheimer projects, Soundcool has shown to be a powerful tool allowing to develop in seconds almost any collaborative music, visual or audiovisual activity/therapy we could imagine. It works with simple interfaces in tablets or smartphones (basically buttons and sliders) and allows remote work. Once a therapy is tested and demonstrated to be successful, it would be developed as a web-based system that is easy for therapists.
Celebrations for the 10th anniversary of Soundcool took place in February 2024, including the Opera Performance Felicità against bullying and harassment. This is our new Soundcool opera project. It is based on an original idea of Professor Nuria Lloret with interactive projections by Stefano Scarani, both from the Soundcool board of directors. The music was composed by us both, featuring live Soundcool electronics, and urban and contemporary dance from the Conservatorio Superior de Danza de Valencia Nacho Duato, with Professor Carmen Giménez as choreographer.
In most live performances with computer music, humans have to follow the computer or even listen to a ‘click track’ over headphones to stay in synchronisation. Roger has developed Accomplice, a system that can follow a keyboard performance, adapting electronics to the tempo and position of the performer. This liberates live performers from playing a strict, pre-determined tempo. We are experimenting with controlling Soundcool with Accomplice so that complex audio effects can be more finely controlled in performance. We are testing the system with Felicità (with pianist Francisco García Verdú, PhD), and some of the talks from the Soundcool 10th anniversary described this new work.
