The best musical ideas flow more easily when there’s a zero-latency connection between mind, body, and instrument. Ever since computers became a common part of the typical household throughout the 1980s and 90s, musicians and programmers alike sought to seamlessly connect body and machine to create a more immersive music experience. The ultimate goal was to make music composition easier so more people could have fun writing and playing music without needing to practice an instrument for years. One group in particular that has pushed the boundaries to an extreme is known as the Opera of the Future.

The Concept

An offshoot of Massachusetts Institute of Technology’s famous Media Lab, located in Cambridge, Massachusetts, Opera of the Future “explores concepts and techniques to help advance the future of musical composition, performance, learning, and expression.” Initially known as the Hyperinstruments project, the group formed in 1986 and first used technology to augment performances of virtuosic instrumentalists. Some early users of Hyperinstruments include Yo-Yo Ma, Peter Gabriel, Prince, and the Los Angeles Philharmonic. Beginning in 1992, the project gradually expanded to create sophisticated, yet intuitive, musical instruments, and more intimate performance experiences, allowing a much wider audience to make use of the most advanced compositional technologies. Presently, the group collaborates with musicians all over the world to create new instruments, performances using advanced technology, and elaborate interactive installations. The achievements of Opera of the Future mostly point back to Tod Machover, the group’s director.

Tod Machover is a composer and technological innovator. The child of a pianist and a computer scientist, it seems he was destined to fuse technology and music. He obtained a Masters degree from Juilliard and began Doctoral studies there, but shifted course and started as a Composer-in-Residence in 1978 at IRCAM in Paris. While in France, he began composing music with synthesizers and began designing what later became known as Hyperinstruments. Machover became a Professor of Music and Media at MIT, in Cambridge, Massachusetts, in 1985. Since then, he has collaborated with countless artists, spurred numerous technological innovations, and gives frequent lectures around the world.

Here are just a few project highlights from over the years.

Hyperinstruments

Machover and his team developed ways to quantify not just what a musician plays, but how they play it. The ability to measure subtle inflections and bodily movements opened up an entirely new realm of interaction between musicians and technology.

The Hyperbow

The earliest Hyperinstrument creation, the Hyperbow is an electronic bow intended for use by professional players of stringed instruments, particularly violin and cello. The Hyperbow uses accelerometers and electromagnetic field sensors to very accurately track its motion, as well as mechanisms that measure the force of the bow against the strings. All the motion and pressure data is transmitted to computers, allowing the musician to supplement their performance with additional sounds, or alter qualities of the resulting sounds. When used in conjunction with a HyperViolin or a HyperCello, which also measure pressure and movement data from the fingerings on the neck, musicians can accomplish numerous layers of sounds and textures with just a single instrument. The composition to the right was performed using only Hyperinstruments. Although there are only three performers (one cellist, violist, and violinist), the Hyperinstruments allow the trio to layer numerous voices and textures. It truly sounds like far more than three people!

The HyperPiano and HyperGlove

Machover’s team also designed an augmented piano, known as the HyperPiano. MIDI data from an electronic keyboard, consisting of velocity (how hard a key is played), duration, and pitch, is processed in a computer and used to manipulate visual components. With this technology, a pianist can supplement and accompany a live performance with a visual element controlled in realtime.

This interaction between body and machine extended to conducting, as well. At the very end of the 1980s, Machover invented what he refers to as the HyperGlove, which allows him to augment an entire orchestral performance from the vantage point of the conductor. He can manipulate dynamics, timbre, and visual components while simultaneously conducting an orchestra. The HyperGlove is a precursor to the Mi.Mu gloves used by Imogen Heap. While initial Hyperinstruments were designed for use by only skilled musicians, Machover wanted to make the technology more accessible to the amateur musician, or even just the average person with no music experience.

Hyperscore

In addition to augmenting live performance, Machover’s team also sought to develop ways to enable the average person to compose an entire score using only a graphic interface. This concept led to the development of Hyperscore toward the end of the 1990s. Hyperscore is a program that essentially allows users to draw music using lines, shapes, colors, and textures. Anybody, including children, can compose elaborate works with no prior music education whatsoever. The Opera of the Future team toured the world with a Hyperscore installation where visitors would collectively compose a score, which was then printed out and performed live. One Hyperscore user, a 10-year-old child with no background in music, composed a piece that was later performed by the Berlin Orchestra in 2001. Hyperscore was released publicly in 2012 and is still available for purchase for PC users. It is the focus of numerous installations and education settings around the world.

Popular Use

Machover’s research team developed other ways to turn music production into a sort of game where even people with no music background could experience the feeling of performing music. One of the most widely known examples of such technology is Guitar Hero, a video game where players use a controller shaped like a guitar to perform popular songs. Guitar Hero and Rock Band, another popular music performance video game, were developed by MIT Media Lab students mentored by Machover. These games use controllers that are shaped and operated like real instruments, particularly guitar, bass guitar, and drum set (pictured right). While they’re drastically simplified compared to the real deal, players need to coordinate their hands to play through the commands given by the game. The motions are very similar to playing guitar; one hand presses buttons, which are the “frets,” while the other hand “strums” a bar representing strings. These games help give people the feeling of performing an instrument without requiring years of practice.

Brain Opera

As a composer and inventor, opera provides the perfect foundation for Machover to experiment with various compositional and technological formats. Opera has always been about pushing the limits of performers and set designers. While most people associate opera with people dressed in viking costumes and ornate performances set in a theater made of marble in 1600s Italy, Machover imagined a futuristic form of opera, featuring robots, advanced technology, and a much deeper level of audience engagement. Thus, the Brain Opera was born.

Premiering in 1996, the Brain Opera set out to bring creative expression to an in-person audience, as well as participants on the internet, by combining numerous interactive elements into one experience. Each performance contains two parts: the exploratory introduction, where in-person attendees get to try out various Hyperinstruments that have been specifically designed for use by the average person, and a more formal 45-minute musical performance led by three conductors, each with “Hyperconductor” abilities. The performance incorporates recordings made by attendees during the introduction, as well as contributions from internet participants. Due to its interactive nature, no two performances are ever the same! After touring the world from 1996-1998, it became a permanent installation at the House of Music in Vienna, Austria.

Death and the Powers

Have you ever heard of an opera featuring robot characters? Probably not, since not many people aside from Machover would think of or even be able to execute such a feat of engineering and orchestration. But where there’s a will, there’s a way.

Death and the Powers is a one-act opera following the story of Simon Powers, a successful and wealthy businessperson approaching the end of his life. He wonders if he can extend beyond his limits by essentially downloading himself into the environment around him to expand his legacy as far as possible. If he completes his transition into his digital realm, known as The System, is he still alive? His family, friends, and business associates must figure out what Simon’s transformation means for their relationships and ability to carry on with their own lives. As technology in the real world progresses, we may eventually have to confront such a mortality dilemma.

Premiering in Monaco in 2010, and in Boston in March 2011, the opera makes use of advanced theatrical technologies. As Simon gradually becomes his environment, his personality manifests itself as expressive stage elements, like a musical chandelier and a literal chorus of robots. The MIT Media Lab refers to this technique as Disembodied Performance. His persona gradually fans out and surrounds the audience.

The opera was critically very well-received. As a result, Machover was a finalist for the 2012 Pulitzer Prize for Music. Death and the Powers is truly a one-of-a-kind experience that pushes performance boundaries and redefines what people consider to be opera.

Schoenberg in Hollywood

One of Machover’s biggest inspirations is the story of Arnold Schoenberg, an influential and controversial Austrian composer. Schoenberg pioneered a new compositional technique, known as atonality, which characteristically creates a lot of harmonic dissonance and tension. Listeners generally find his music somewhat discomforting and unusual.

Although Schoenberg had established himself in Germany at the Berlin Conservatory and converted away from Judaism, he was forced to flee the Nazi regime in 1935. He ended up in Hollywood, California, where he became friends with some of the biggest names in show business at the time, like George Gershwin and Charlie Chaplin. He was eventually introduced to a gentleman named Irving Thalberg, who headed MGM Pictures at the time. Thalberg wanted Schoenberg to score an upcoming movie, titled The Good Earth. Due to his personal experiences with overcoming some of the worst humanity has to offer, he was very conflicted about scoring the film. Machover sought to compose an opera around based around Schoenberg’s story.

The opera premiered in November 2018 at the Paramount Center at Emerson College in Boston. In order to meet the demands of the story, the stage setup is incredibly advanced, making use of cutting-edge visuals and a set that can rapidly change appearance. The sound design takes on a life of its own. Dozens of microphones and speakers, placed in very specific locations, are scattered throughout the theater. This incredibly nuanced take on surround sound allows the ensemble to effectively move around the theater, changing size, shape, and texture. But Machover really wanted to create more of a subtle effect. The technology isn’t meant to be the focus of the production. It simply allows the production team to best execute the vision. It’s quite likely the most technologically advanced opera ever performed. For a glimpse into the production, check out this video.

MIT’s Opera of the Future’s accomplishments are numerous and far-reaching, encouraging people of all ages, backgrounds, and musical experiences to participate in the creative process of composing and performing music. As they continue to advance technologies, their developments are changing the way people learn, express themselves, and interact with one another. Click here to explore previous and upcoming projects by Opera of the Future.

Schedule a 50%-Off Trial Lesson

Want to better understand the underlying concepts that go into futuristic musical instruments and performances? Schedule a trial lesson to get closer to your goals!

Opera of the Future: Redefining Music Performance