Zenph - A Pragmatic Project - Live In Concert Apr 2005

A Pragmatic Project: Live In Concert by Andy Hunt, The Pragmatic Programmers, LLC. All too often we hear of well-intentioned software projects that crash and burn due to a long and familiar list of common pitfalls. But every now and then we come across a truly Pragmatic Project where a software development team tackles a monstrously complex problem, does all the right things that we've written about in our books and articles and emerges triumphant. From version control to unit testing (including a novel approach for fuzzy testing of complex data) to automation (including a webcam), we'll look at a team that does it right. First, let me tell you about the project itself. Early in March 2005, the folks at Zenph Studios invited me to a nearby recital hall to hear the famous pianist Glenn Gould play solo piano on a 9-foot concert grand, live for an audience of about 30-40 people. But here's the twist: the piece was played exactly as he played it on a 1955 recording, and Gould himself died over twenty years ago. Was this a pragmatic séance in operation? No. Player piano rolls? No. All the team had to go on was a well known but scratchy mono recording made in 1955, and here was a live piano replaying the concert note for note. And just to prove how identical the performance was, our host, John Q. Walker (formerly of Ganymede, now president of Zenph) played a recording of the new performance in one speaker and switched back-and-forth to the original mono recording in the other speaker. The sound was identical, except for the scratches, clicks, and pops on the mono recording. From that old vinyl mono recording, the team at Zenph Studios analyzed the audio waveforms and extracted every nuance of the original performance: articulation, technique, pedals; everything needed to recreate the performance. After analysis, this information is then stored in a high-definition MIDI file (which is a quite a bit more sophisticated than the more familiar, and now dated, MIDI standard). Finally, this data is fed to a freshly-tuned and voiced souped-up Disklavier Pro piano (of which only several currently exist in the world). This is not the same Disklavier you might hear in a Marriott or Embassy Suites, endlessly banging out Misty for hapless passersby. This is a high-performance, cutting-edge instrument, capable of reproducing world-class, concert pianist performances.

Challenges Which brings us to the challenges faced by the Zenph team. As with any bleeding-edge technology, the Disklavier Pro came with no technical specifications describing the high-definition MIDI format it uses to drive the instrument. As Zenph A Pragmatic Project: Live in Concert Copyright © 2005 The Pragmatic Programmers, LLC April 2005 Page 1

soon discovered, there were gaps in the data as well. Needless to say, there were no supporting libraries to deal with high-definition MIDI files, either. Then there's the audio side of it. How do you determine which notes were hit, with what finger, articulation, the position of the piano pedals and all the rest of that when it's mixed up in a monophonic audio track with all the ambiance, echo, and crowd noise? How on earth would you even unit test this sort of thing? Oh, and modern startup companies, unlike the Foosball-equipped wastrels of the dot-com era, can't afford to just throw money at a problem, so you need to adopt a very pragmatic approach. Which they did.

The Methodology The Zenph team consists of three developers and one expert user. Yes, the user is very much part of the team, and in fact, drives the development effort, but more on that a bit later.

Setup The team is geographically dispersed across the U.S., connected by a virtual private network (VPN). That's less surprising these days than in years past (and in fact is the rule on virtually every project we've worked on), but none of our projects have ever featured a fully-automated recording studio complete with webcam! With this setup, the developers can feed high-definition MIDI files to the Disklavier Pro, watch and record the results using the studio microphones, and download the resulting audio file to wherever they happen to be. Version control should be the first thing any project implements, and Zenph started with Perforce from Day One of the project. Unlike many unsuccessful projects, there was never even a question about using a mature, robust version control system for development. Again, unlike many others, they put everything under version control, from PowerPoint presentations for potential investors to their Articles of Incorporation and so on.

Testing Now we get to the interesting part. How on earth do you unit test in this sort of environment? JUnit, for example, doesn't have an assertion for "does this audio file sound close to this other audio file?" Their goal is to generate high-definition MIDI files that sound like those created when real performers play at the piano. Here's their main coding/testing workflow: •

Start by creating a high-definition MIDI file using the given Disklavier Pro technology (that is, physically play a note or two on the piano keyboard). A Pragmatic Project: Live in Concert Copyright © 2005 The Pragmatic Programmers, LLC April 2005 Page 2

•

Now drive the piano using that MIDI file, and record the results through the air using regular microphones in the studio to produce a WAV-format audio file.

•

Process the audio file using the software under test to produce a MIDI file again.

•

Compare the original MIDI file to the regenerated one.

Ah, but there's the tricky part. Unlike standard MIDI, these high-definition MIDI files are built with around 10 attributes per note, all shifting and sliding around in time as well. Using simple assert statements as one would find in JUnit, CPPUnit and so on won't cut it: the high-definition MIDI files don't have to be exact (and in fact won't be); they have to sound the same. Now it turns out that some perceptual acoustics come into play. The human ear is very discriminating when it comes to timing, for instance. A difference of even 2-3 milliseconds is very noticeable in the right context. So they came up with the concept of a Grader. The Grader is a bit of testing framework that compares high-definition MIDI using a set of heuristics, examining all 10 attributes per note, and judging the "performance.” This gave the team a way to quantify the end goal of does it sound right? As we describe in our Pragmatic Workshops, you first have to know what "done" means in order to actually be done. Without an objective measure of success, this team could have floundered internally for months arguing whether they were converging on a solution or not, or worse, delivering product that the end user wouldn't accept. But not only did the team solve that problem, they took the idea one step further and made the boss write the Grader. In other words, they made the project manager define success in a quantifiable, achievable manner. That alone can make the difference between a successful project and a death march. It's also important to note that the Grader and its heuristics didn't spring into life fully complete. They added heuristics to the Grader incrementally as development progressed: a 3ms discrepancy is better than a 300ms one, and one isolated discrepancy is better than five in a row, and so on. They tuned and grew the grader as they went, slowly but steadily converging on the goal. Some teams would want to start testing with a Rachmaninoff piano concerto or somesuch, and that's a huge mistake. You always need to start with small, isolated unit tests before moving on to more advanced functional or acceptance testing. In this case, the first six months or so of unit tests comprised some beautiful piano solos made of just one note. Just one single note at a time, mapping out the full range of the instrument. It was quite a while before the unit tests got the software to the point where they could try "Mary Had A Little Lamb," and quite a while after that before poor Mary got any rhythm.

A Pragmatic Project: Live in Concert Copyright © 2005 The Pragmatic Programmers, LLC April 2005 Page 3

Requirements

Agile methodologies recommend that you have an expert as part of the development team, so that you can get rapid feedback as the software matures and quick decisions on gray areas. But there's a possible down side to such intensive user involvement. As part of the development team, the user may begin to lose sight of the distinction between their true needs and various implementation strategies. As with any project, the challenge is in drilling down to the user's true need without being distracted by their view of possible implementations. The Zenph team has an expert user---now completing a doctorate in Piano, no less. His close association with the team shortened the feedback gap and let the team move with great speed at satisfying the requirements.

Developers working in the recording studio.

The Technology Different programming languages are better suited for certain applications than others; the trick is to use the right tool for the job. Telecom and audio apps tend to favor C++, so their production-level audio analysis software is written in C++. But that's not the end of the story---they don't use C++ for everything. For rapid prototyping of low-level algorithms, the developers use a scripting language (they happen to use Perl; other teams we know use Ruby or Python). They use the same scripting language to power the automated build and test, and for controlling and coordinating remote-control access to the studio and equipment. On the C++ side, they've continued the practices that John and the developers used at Ganymede: lots of good instrumentation in the code itself. This includes custom runtime assertions with full call-stack reporting and so on. As to the age-old debate as to whether you should leave these assertions enabled at runtime, the Zenph team comes down heartily on the side of enablement. Since they are deploying this software in a service-bureau A Pragmatic Project: Live in Concert Copyright © 2005 The Pragmatic Programmers, LLC April 2005 Page 4

model, there's no downside to displaying very technically detailed assertion failures at runtime (software designed for your grandmother, on the hand, may need to adopt a somewhat less threatening posture). Another enduring question that plagues many teams is "build or buy?” When is it most advantageous to build custom software, and when is it better to just buy it? As developers, we tend to prefer to build our own stuff all the time (pragmatic author Venkat refers to this as "Resume-Driven Design"). The team struck a good balance, leveraging open source, commodity-level functionality where possible. Items such as path and extension handling, threading libraries, GUI widgets, and so on were ably handled by libraries from boost.org, the wxWidgets set, and others. They wisely reserved custom crafting where it was most needed, including their core audio algorithms and components such as the MIDI data pretty printer. Now printing out binary MIDI data in a nicely formatted, human-readable form may sound like a frivolous extra, or at least something one could get off the shelf. But as the team explained to me, they didn't want some third party piece of software to mask, distort, filter, or otherwise lie about the data. They needed to know precisely what was going on, with no surprises. Too often, we take critical diagnostics on faith, only later to realize that the information wasn't accurate or complete.

Team Work The most important ingredient of any project is the people. Not only do you need good people to begin with, you need people who can work together effectively. On larger teams, you can afford to have some trainees aboard---folks who aren't at the top of their game yet, but aspire to grow in that direction. On small teams, you may not have that luxury. In this case, with such a small team, each member is a highly-skilled professional. To get maximum skill coverage, each specializes in separate, non-overlapping areas of expertise: the high-definition MIDI format, audio processing and acoustics, infrastructure and architecture, and concert piano performance. Each team member is good at prioritizing their own tasks and maintains their own to-do lists. But despite this autonomy, they still hold weekly virtual meetings to share progress and status and coordinate their plans.

A Pragmatic Project: Live in Concert Copyright © 2005 The Pragmatic Programmers, LLC April 2005 Page 5

Developer team meeting. To leverage their computing environment and maximize mobility and interoperability, all their development work is done on laptops (larger servers or grids will be used in production). Even better, all the laptops are hardware-compatible IBM ThinkPads. This allows the team members who are collocated to share parts more easily in case of emergency, and makes issues such as OS and driver patches much easier to manage. But while the development hardware platform is standardized, the software isn't. There is no mandated IDE--nor should there be. Different people will be more productive in different IDE environments, and these team members tend to prefer powerful stand-alone editors in lieu of bloated, slow, all-encompassing IDE's (note the complete lack of editorial bias here). With this technology and methodology in hand, this small team was able to create groundbreaking software in a relatively short space of time. Let's take a quick recap of the salient points of their approach.

Lessons Learned What can we learn from this team? Just that the basics on which we continue to publish and lecture make a big difference. Your team could be a success or yet another dismal statistic, it's up to you. Here the practices observed by the Zenph team that we think are most important. •

Start with good people. If you're lacking in certain skills or experience, invest in training or workshops to help bring everyone up to speed (if you've got trainees you need to bring along, make sure they have a chance to learn while keeping them out of harm's way on the critical path). If you're faced with choosing between a mediocre team that's collocated vs. a skilled team that's geographically dispersed, choose the skilled team and invest in a decent VPN.

•

Everything under version control. We're getting kinda tired of beating this drum, but many teams still either don't use version control at all or do so improperly. If your version control system is getting in your way or slowing you down, you need to replace it or learn how to use it properly. By the way, my address book is under version control, as well as my daily to-do list.

A Pragmatic Project: Live in Concert Copyright © 2005 The Pragmatic Programmers, LLC April 2005 Page 6

•

Define success. The project manager or sponsor needs to define what "done" means for the team. And not just in some flowery mission statement. Give developers quantifiable and achievable goals and they'll meet them.

•

Start small. Tracer-Bullet development is all about starting with a full framework, but with all the functionality stubbed out. Get the small, fundamental basics right first, operating within a sketch of the larger architectural context.

•

Test. There's no excuse not to unit test. If the code is "too hard" to test, then it's likely too hard to write. Well-designed code is easily testable; poorly-designed code is not. Don't be afraid to create your own test framework or supporting code (like the Grader) if needed.

•

Learn from the user. Listen closely to the user---their expert knowledge should be driving the requirements. But their suggestions should not determine the architecture and design, that's the developer's job. Always look past the user's stated requirement to discover their real need.

•

Leverage technology. Use open source where it makes sense. Build proprietary software where it makes sense. Don't be dogmatic in either case. Fancy or expensive IDE's may be a great help, or they may slow the team down. Developers usually know which is true for them; make sure the team honors that.

•

Teams build products. To build a great product, you need to also build a great team. Whether your team is composed of novices or experts make sure everyone is allowed to work appropriate to their skill level, and can all work together well (that's all easier said than done, of course, but that's a topic for another book).

Looking Forward John Walker says there are a great many historic recordings that were never released to the public due to minor recording problems---an out of tune string, someone in the audience coughing up a furball, that sort of thing. But once converted into the high-definition MIDI domain, the performance can be edited: a blown note fixed, technique adjusted to match the response of modern equipment, and so on. When recording the Disklavier, there doesn't need be an audience present. That means the recording engineer doesn't have to contend with squeaks from the piano bench, the air-conditioning suddenly kicking on, or anything of that nature. And even if some noisy event did ruin the take, it's trivial to restart the high-definition MIDI playback and re-record. Thanks to well-written software that meets the requirements, this technology may well end up having vast repercussions for historians, performing artists, and fans alike. That's what we call a successful project.

http://www.pragmaticprogrammer.com/articles/zenph/ A Pragmatic Project: Live in Concert Copyright © 2005 The Pragmatic Programmers, LLC April 2005 Page 7