Headphone Roundup - Overview, Methodologies, and Scorecard
by Wayne Myers
by Wayne Myers
Note: If the following tables do not fit in your browser window, use Ctrl+"-"
("Control" and "minus" keys together) to step their size smaller.
Use Ctrl+"+" to step larger, Ctrl+"0" to return to default size.
Type: Full-Sized, Open, Around-Ear
No code has to be inserted here.
Type: Full-Sized, Closed, On-Ear
No code has to be inserted here.
Frequency Response Curves and Sound Demos
Beyerdynamic DT 880 Premium 250-Ohm
Sennheiser HD 600
Overview and Methodologies
Introduction - Two Facts About Headphones
Our first challenge is to convince you that the world needs another collection of headphone reviews. Consider two basic facts:
Fact number one: Headphones and headphone listening preferences tend to get very personal. They fit on your head and cover your ears. The music you hear is all inside your head. They separate you from your surroundings, allow you to enjoy your chosen tunes without bothering others at home, in the car, on subways, buses, airplanes, at work. This personal, inside-the-head experience is hard to share, leaves us searching for words, and the ones we come up with can end up sounding downright ridiculous.
Fact number two: Headphone characteristics are hard to measure. How do you measure what is going on inside a person's ear, way down at the end of the ear canal, at the eardrum? How do you put numbers on it? Headphone measurements involve expensive calibrated heads and ears with special microphones and special measurement transfer functions. Most reviewers have no choice but to work in the realm of the subjective.
These two facts make the headphone a somewhat unique animal in the world of audio, where there are already plenty of reasons to disagree about what sounds good and what does not. As luck would have it, they also present us with an opportunity.
The Goal - A Different Kind of Headphone Review
This project was born during the summer of 2012 when I was visiting a fellow audiophile family member who had several sets of nice headphones I was not familiar with. In no time I was listening, comparing, taking notes, and before I knew it the idea for a quantifiable comparison method was being born. The approach centered on the following idea: Why not take subjective listening tests and feed their results into an objective scoring system for comparing headphones?
Subjective listening tests can be performed by anyone, and the only standardization for the results ends up being, "I trust her or his opinion," much like the way we end up trusting a particular movie reviewer. So why not develop a system with very specific listening tests, right down to the test track and what to listen for, taking the mystery out of the testing and giving direct comparisons between different headphone models?
The Objective-Subjective Headphone Review - Details of the Approach
A spreadsheet was quickly born, test tracks were chosen, along with test criteria, and I was well on the way to having an objective–subjective test method which anyone could apply. When the opportunity presented itself here at the Home Theater Shack to give this project a home, I jumped at the chance.
I decided early on to put headphones across a spectrum of designs on an equal footing. Rather than dividing them by price range and then comparing qualities within those ranges, the approach here is to compare all models directly, regardless of price. Why? Sometimes there are surprise models that defy their price range. If a $100 headphone manages to sound as good as a $300 headphone - I have not seen it yet, but there are those that come close - then they should be allowed to compete directly. Each of the various listening tests, then, results in a score, weighting factors are applied, numbers get crunched and combined, and a final performance score is arrived at for each model of headphone.
As you will see below, six listening categories and two non-listening categories have been identified. Test tracks have been specified, listening criteria have been detailed, and a scoring method is outlined for each. There is a formula for combining scores to come up with a category score, and finally the grand Overall Performance Score pops out at the end. The scores are all entered into a comparison matrix, the Headphone Roundup Scorecard, which makes it easy to directly compare the listening characteristics of different models.
- These tests are all comparative in nature. Reference headphones for each category are compared, often switching back and forth several times, in determining which scores to assign.
- The test tracks mentioned are personal favorites. In the case of the two chosen for high-frequency imaging tests, they are by far the best I have found for the task; nothing else comes close. In all other cases, there are many possible candidates. The formulas are set up to work with any suitable test tracks.
- For the Imaging, Clarity, and Speed tests, it can take some time to catch just the right details necessary to make the evaluations. There is no need to rush this process. For the Soundstage and Frequency Response tests, the ear acclimatizes to whatever it is hearing after a minute or two, so these evaluations should be made fairly quickly, in a minute or less.
- Additional descriptors often used in headphone reviews include words like "detail," and "resolution," and numerous others. It is my opinion that the categories included cover the headphone characteristics that matter the most, and include the characteristics that these other descriptors try to address.
- A good quality headphone amplifier is a must. The Firestone Audio Cute Beyond Headphone Amplifier with Class A output stage and the FiiO E10 USB DAC Headphone Amplifier were used for these tests. (It is highly recommended that you use a headphone amplifier with near-zero (under one ohm) output impedance. Both of the amplifiers used meet this requirement. Many headphones do not need this to sound their best, but SOME DO.)
- Test tracks should be lossless or highest-quality MP3 files (320 Kb/s). The high-frequency imaging and clarity tests must be done with lossless files.
- All headphones were either used or burned in for 100 hours or more before evaluation.
- A tip of the hat to Tyll Hertsens, the man behind a good part of the useful headphone measurement data available on the web.
Categories for Evaluation and Scoring
Eight evaluation categories are used in this objective-subjective approach, all feeding into a single Overall Performance Score. The first six categories are listening categories. Under the Measurement Methods button below you will find detailed descriptions of what to listen for and how to score a set of headphones in each of these categories:
- Imaging - The apparent ability for the listener to precisely locate the sound of a voice or instrument in space.
- Soundstage - The ability for headphones to create the illusion that the instruments and voices in the music are stretched across an imaginary stage in front of the listener.
- Clarity - Freedom from distortion.
- Speed - The ability to handle tones with a fast or sharp attack time.
- Frequency Response - The evenness of tonal balance. Four general types of voicing, or frequency-response profiles, are defined.
- Overall Listening Experience - How the headphones sound in general, how enjoyable they are to use.
Headphone Roundup Scorecard
The Scorecard used to be right here. Now it is at the top of this document.
Note: This thread remains open for discussion about the Overview, the Scorecard, and the evaluation methods used. Please post comments about an individual headphone model or review in its own discussion thread.
There are a lot of headphones out there, and a lot of headphone reviewers. The purpose of all this is to give you a different kind of tool to help you decide, based on these objective-subjective scores, which headphones you might like and want to buy. Yes that is an ambitious goal. Please let me know how well it works for you. Also remember, I gave this my best shot, based on my experience and my best judgments, but your own ears will be your own best judge. I welcome your constructive feedback, but alas, of course I cannot guarantee satisfaction based on the scores - short of joining you in a Venetian mind meld. That would cost extra. I do sincerely wish you the best in your headphone listening and purchasing decisions.
May your headphone listening sessions be many and awesome!
by Wayne Myers
- 1-18-2013: Explanation in Imaging scoring method, definition of "Drivable with portable media devices."
- 2-1-2013: Explanation in Imaging scoring method, definition of "Drivable with portable media devices."
- 3-22-2013: New format for Scorecard; Scorecard now comes first in document; photos, frequency response curves, and frequency response demos added; links to Reviews now part of Scorecard.