Joined
·
5,013 Posts
Scorecard
Note: If the following tables do not fit in your browser window, use Ctrl+"-"
("Control" and "minus" keys together) to step their size smaller.
Use Ctrl+"+" to step larger, Ctrl+"0" to return to default size.
Type: Full-Sized, Open, Around-Ear
No code has to be inserted here.
Type: Full-Sized, Closed, On-Ear
No code has to be inserted here.
Frequency Response Curves and Sound Demos
Beyerdynamic DT 880 Premium 250-Ohm
Pioneer SE-A1000
Sennheiser HD 600
Velodyne vTrue
Overview and Methodologies
Introduction - Two Facts About Headphones
Our first challenge is to convince you that the world needs another collection of headphone reviews. Consider two basic facts:
Fact number one: Headphones and headphone listening preferences tend to get very personal. They fit on your head and cover your ears. The music you hear is all inside your head. They separate you from your surroundings, allow you to enjoy your chosen tunes without bothering others at home, in the car, on subways, buses, airplanes, at work. This personal, inside-the-head experience is hard to share, leaves us searching for words, and the ones we come up with can end up sounding downright ridiculous.
Fact number two: Headphone characteristics are hard to measure. How do you measure what is going on inside a person's ear, way down at the end of the ear canal, at the eardrum? How do you put numbers on it? Headphone measurements involve expensive calibrated heads and ears with special microphones and special measurement transfer functions. Most reviewers have no choice but to work in the realm of the subjective.
These two facts make the headphone a somewhat unique animal in the world of audio, where there are already plenty of reasons to disagree about what sounds good and what does not. As luck would have it, they also present us with an opportunity.
The Goal - A Different Kind of Headphone Review
This project was born during the summer of 2012 when I was visiting a fellow audiophile family member who had several sets of nice headphones I was not familiar with. In no time I was listening, comparing, taking notes, and before I knew it the idea for a quantifiable comparison method was being born. The approach centered on the following idea: Why not take subjective listening tests and feed their results into an objective scoring system for comparing headphones?
Subjective listening tests can be performed by anyone, and the only standardization for the results ends up being, "I trust her or his opinion," much like the way we end up trusting a particular movie reviewer. So why not develop a system with very specific listening tests, right down to the test track and what to listen for, taking the mystery out of the testing and giving direct comparisons between different headphone models?
The Objective-Subjective Headphone Review - Details of the Approach
A spreadsheet was quickly born, test tracks were chosen, along with test criteria, and I was well on the way to having an objective–subjective test method which anyone could apply. When the opportunity presented itself here at the Home Theater Shack to give this project a home, I jumped at the chance.
I decided early on to put headphones across a spectrum of designs on an equal footing. Rather than dividing them by price range and then comparing qualities within those ranges, the approach here is to compare all models directly, regardless of price. Why? Sometimes there are surprise models that defy their price range. If a $100 headphone manages to sound as good as a $300 headphone - I have not seen it yet, but there are those that come close - then they should be allowed to compete directly. Each of the various listening tests, then, results in a score, weighting factors are applied, numbers get crunched and combined, and a final performance score is arrived at for each model of headphone.
As you will see below, six listening categories and two non-listening categories have been identified. Test tracks have been specified, listening criteria have been detailed, and a scoring method is outlined for each. There is a formula for combining scores to come up with a category score, and finally the grand Overall Performance Score pops out at the end. The scores are all entered into a comparison matrix, the Headphone Roundup Scorecard, which makes it easy to directly compare the listening characteristics of different models.
Notes
- These tests are all comparative in nature. Reference headphones for each category are compared, often switching back and forth several times, in determining which scores to assign.
- The test tracks mentioned are personal favorites. In the case of the two chosen for high-frequency imaging tests, they are by far the best I have found for the task; nothing else comes close. In all other cases, there are many possible candidates. The formulas are set up to work with any suitable test tracks.
- For the Imaging, Clarity, and Speed tests, it can take some time to catch just the right details necessary to make the evaluations. There is no need to rush this process. For the Soundstage and Frequency Response tests, the ear acclimatizes to whatever it is hearing after a minute or two, so these evaluations should be made fairly quickly, in a minute or less.
- Additional descriptors often used in headphone reviews include words like "detail," and "resolution," and numerous others. It is my opinion that the categories included cover the headphone characteristics that matter the most, and include the characteristics that these other descriptors try to address.
- A good quality headphone amplifier is a must. The Firestone Audio Cute Beyond Headphone Amplifier with Class A output stage and the FiiO E10 USB DAC Headphone Amplifier were used for these tests. (It is highly recommended that you use a headphone amplifier with near-zero (under one ohm) output impedance. Both of the amplifiers used meet this requirement. Many headphones do not need this to sound their best, but SOME DO.)
- Test tracks should be lossless or highest-quality MP3 files (320 Kb/s). The high-frequency imaging and clarity tests must be done with lossless files.
- All headphones were either used or burned in for 100 hours or more before evaluation.
- A tip of the hat to Tyll Hertsens, the man behind a good part of the useful headphone measurement data available on the web.
Categories for Evaluation and Scoring
Eight evaluation categories are used in this objective-subjective approach, all feeding into a single Overall Performance Score. The first six categories are listening categories. Under the Measurement Methods button below you will find detailed descriptions of what to listen for and how to score a set of headphones in each of these categories:
- Imaging - The apparent ability for the listener to precisely locate the sound of a voice or instrument in space.
- Soundstage - The ability for headphones to create the illusion that the instruments and voices in the music are stretched across an imaginary stage in front of the listener.
- Clarity - Freedom from distortion.
- Speed - The ability to handle tones with a fast or sharp attack time.
- Frequency Response - The evenness of tonal balance. Four general types of voicing, or frequency-response profiles, are defined.
- Overall Listening Experience - How the headphones sound in general, how enjoyable they are to use.
- Comfort
- Design
Measurement Methods
Imaging:
Imaging is the apparent ability for the listener to precisely locate the sound of a voice or instrument in space. Tight imaging, especially with a wide soundstage, can lead to the impression of voids between other well-localized sounds. This effect can take some getting used to. However, tight imaging can also be a transcendent quality of headphones in making a voice or instrument or sound seem in-the-room real.
- My reference headphone: Sony MDR-V6
- Test track and what to listen for:
- Note: Small amounts of individual high-frequency hearing loss in one ear or the other can cause apparent image smearing on either of the high-frequency test tracks.
- Ain't It A Shame, the B-52's - very high-frequency imaging - Cindy's vocals have a glossiness to them which shows up in the octave between 7 kHz and 14 kHz. All aspects of the sounds of her voice should remain precisely centered. Especially listen for smearing on the "S" sounds, the sibilants. With headphones that do not image well parts of the S's will appear to smear left or right, or will be all over the place, difficult to locate precisely. The harmonica tones also contain complex harmonics which will seem hard to localize precisely if imaging is weak. Precise imaging keeps all these sounds perfectly together as a pinpoint source, no smearing or imprecision, no sense of broadness or vagueness, no wandering of any part of that sound on any notes. Some headphones do not have a high enough frequency response for that glossiness to be heard. In that case, they are not penalized since the imaging is not degraded, and the formula is modified to leave out that one factor.
- Reasons Why, Nickel Creek - high-frequency imaging - Sara's lead vocals contain strong sibilants, falling in a slightly lower frequency band than with the previous track. They will remain precisely centered with no sign of smearing on headphones that image well.
- Ode To A Butterfly, Nickel Creek - general imaging - With precise imaging, it is as though you can precisely locate in space the point where pick meets string for the mandolin and guitar throughout this piece. The fiddle will also will seem pinpoint precise throughout its range. With the standup bass, listen for the string buzz near the end of the song, its location should appear laser sharp.
- Scoring:
- The result of each of the three Imaging tests is given an integer score between 0 and 10. I settled on assigning integer evaluation scores so I wouldn't spend forever splitting hairs with fractional scores. All calculations thereafter are rounded to the nearest 1/10.
- 10: Perfect imaging, no broadening, smearing, spreading, wandering, or lack of distinction whatsoever, laser sharp throughout.
- 8 - 9: Strong imaging but with slight broadening or occasional small amounts of smearing.
- 4 - 7: Fair imaging, generally lots of smearing, image somewhat broad and indistinct, may appear to spread out or move around on certain tones.
- 0 - 3: Poor imaging very broad, indistinct, and unstable.
- Formula applied: (IM.very.hf + IM.hf + (2 x IM.general)) / 4, rounded to the nearest 1/10th. For headphones with weak highs, the formula is modified to:
(IM.hf + (2 x IM.general)) / 3 - Overall Weighting: x2
Soundstage:
Soundstage describes the ability for headphones to create the illusion that the instruments and voices in the music are stretched across an imaginary stage in front of the listener.
One of the potential stumbling blocks for headphone listening is that the soundstage tends to be located inside the listener's head. The brain perceptually tries to move it back outside of the head and in front of the listener, because that is where the brain thinks sounds like that should be coming from. This psycho-acoustical effect occurs quite naturally for some listeners and not at all for others. Imaging for headphones will not sound the same as imaging can from loudspeakers. Speakers can deliver a very wide and deep two-dimensional soundstage, since the listeners' ears each hear sounds from both speakers. Headphones are generally limited to giving a one-dimensional soundstage, which can seem very wide but will usually have no depth other than the illusion of depth resulting from different volume levels for different sounds.
The key here is the perception of how natural the soundstage feels. With some headphones, the soundstage feels tense or uncomfortable or somehow false. Better headphones deliver a soundstage that is easy to imagine as open, natural, and real.
- My reference headphone: AKG K701
- Test tracks and what to listen for:
- Ode To A Butterfly, Nickel Creek - The soundstage for this Bluegrass track should be wide and natural, may actually seem to have voids between the instruments.
- Fanfare/You Know It, Tower of Power, Direct Plus - The big Funk band should appear evenly spread across a wide, natural soundstage.
- Scoring
- The result of each of the Soundstage tests is given an integer score between 0 and 10.
- 10: Perfect soundstage, wide, natural, convincingly real, totally comfortable.
- 8 - 9: Good soundstage, but not quite convincing or natural for certain types of music.
- 4 - 7: Fair soundstage, generally feels somewhat unnatural, difficult to place instruments and voices.
- 0 - 3: Poor soundstage, very uncomfortable and unnatural.
- Formula applied: SS = (SS1 + SS2) / 2, rounded to the nearest 1/10th.
- Overall Weighting: x2
Clarity:
Clarity relates most closely to freedom from distortion. Harmonic distortion is fairly easy to measure in the lab and is commonly stated by the manufacturer, but under listening conditions discerning clarity can be more difficult. Instruments with complex harmonics can be useful, especially on tracks with a lot of other things going on. My favorites are loud rock tracks with well recorded cymbals. Some of these tests have to be cranked up quite loud to be effective. Never dangerously loud, but well into the 90 dB listening range, only for a minute or two at a time.
- My reference headphone: Sony MDR-V6
- Test tracks and what to listen for:
- Bland Street Bloom, SikTh - cymbals, loud track - The complex harmonics of these cymbals should be very clear and distinct. With poor clarity, the cymbals will sound like shaped noise with no distinct harmonics.
- Bland Street Bloom, SikTh - bass, loud track - The bass should be very clear, not distorted.
- Disruptr and Heaven Send, Devin Townsend - cymbals, quiet track - During the quiet passages, the complex harmonics of these cymbals should be very clear and distinct.
- Disruptr and Heaven Send, Devin Townsend - cymbals, loud track - During the loud passages, the complex harmonics of these cymbals should be very clear and distinct. Poor clarity here will leave the cymbals sounding like shaped noise with no sign of distinct harmonics. With good clarity, those harmonics might almost sound out of place in the midst of the other loud, distorted instruments.
- Beethoven, 7th Symphony, Second Movement, Chicago Symphony, F. Reiner conducting - strings, loud track - When the movement reaches its loudest point, the strings should remain clear and natural sounding, no edge or impression of there being "something else" going on there.
- Scoring
- The result of each of the Clarity tests is given an integer score between 0 and 10.
- 10: Perfect clarity, bass is always smooth and distinct, complex cymbal harmonics are always clear and easily discernible, strings always smooth and natural, none of these ever distorted, no edge or impression that something is not quite right about it.
- 8 - 9: Good clarity, but occasionally a bit messy or slightly distorted, not quite perfect.
- 4 - 7: Fair clarity, generally somewhat messy sounding, often distorted during louder passages.
- 0 - 3: Poor clarity, very messy sounding, quite distorted most of the time.
- Formula applied: CL = (CL.cymbal.1.loud + CL.bass.1.loud + CL.cymbal.2.quiet + CL.cymbal.2.loud + CL.strings.loud) / 5, rounded to the nearest 1/10th.
- Overall Weighting: x2
Speed:
Speed defines the ability for headphones to handle tones with a fast or sharp attack time. "Fast" headphones handle the onset of those sounds cleanly keeping them feeling crisp and tightly defined, getting a sense of impact to each note. Sometimes you will hear the word "punchy" used in this context. Slower headphones give an impression of looseness or mushiness at the beginnings of those sounds.
- My reference headphone: AKG K701
- Test tracks and what to listen for:
- Beyond The Blue, Beth Nielsen Chapman - deep bass - Focus: deep pulsing drumbeat.
- Good Stuff, the B-52's - lows - Focus: bass guitar.
- Inchworm, Battles - mids - Focus: drums, especially toms.
- Pulse, Todd Rundgren - upper mids - Focus: xylophone.
- Healing Part I, Todd Rundgren - highs - Focus: triangle.
- Scoring
- The result of each of the Speed tests is given an integer score between 0 and 10.
- 10: Perfect speed, every test example is delivered with crisp, tight impact.
- 8 - 9: Good speed, but occasional instances that are delivered slightly soft or mushy, not quite perfect.
- 4 - 7: Fair speed, generally somewhat soft, mushy delivery of sounds that should have an impact.
- 0 - 3: Poor speed, very soft and mushy delivery.
- Formula applied: SP = (SP.deep.bass + SP.lows + SP.mids + SP.upper.mids + SP.highs) / 5, rounded to the nearest 1/10th.
- Overall Weighting: x2
Frequency Response:
Frequency response is probably the single most important quality contributing to headphone performance. That said, it is somewhat difficult to score because there is no one type of frequency response that all listeners accept as the ideal. My own preference is for a fairly flat frequency response, but it would be unfair to score all headphones based on that one narrow-minded requirement.
We will work with four frequency response profiles which hopefully cover in a reasonably unbiased way the different voicings given to headphones today. They are:
- Frequency Response Profiles:
- Flat - Preferred by many listeners and especially by those who use headphones as part of an engineering, mixing, or mastering process. This profile allows for small amounts of low frequencies or high-frequency boost.
- Tilted - A fairly flat but tilted response curve, downward in the high-frequency direction. This is a more laid-back profile which is often preferred for general listening as it can be less fatiguing for long listening sessions. This profile often includes a certain amount of "scooping" as described next.
- Scooped - A region of the frequency spectrum, usually in the upper midrange, is pulled back, or "scooped" out of the curve. If not too extreme, the ear doesn't miss those frequencies since they are in a region where the hearing is more sensitive anyway, and the result is a laid-back sound without loss of the highest frequency range, containing the "detail" for many sounds.
- Emphasized Bass - This profile is often preferred for popular music listening, with low frequencies highly emphasized, sometimes 20 dB or more.
- My reference headphone: AKG K701 (Flat Profile), Sennheiser HD 600 (Tilted Profile)
- Test tracks and what to listen for:
- Revolution Earth, the B-52's - This track has all ranges quite equally represented. It should sound bright and lively, not harsh, with solid bass tones, but not overly strong.
- Glory Bound, Wailin' Jennys - This track contains deep standup bass tones which should give a strong, deep thump without being overbearing.
- Fanfare/You Know It, Tower of Power, Direct Plus - All of the instruments and voices in this direct-to-master recording from the early 80s were given minimal treatment, kept sounding as simple and natural as possible. Pay special attention to the horn section. Headphones with an overly-aggressive "scoop" out of the upper mids may sound fine for many instruments, even for male and female vocals, but will sound fakey or downright bad with saxophones or other horns.
- Scoring
- The result of each of the Frequency Response tests is given an integer score between 0 and 10. Over-emphasis of a frequency range stands out more to the ear than under-emphasis, and therefore is scored more negatively.
- Each of the following frequency ranges is evaluated while listening to each track:
- Deep bass
- Lows
- Mids
- Upper mids
- Highs
- When evaluating frequency response, there has to be a reference band. This will usually be the Mids, but with some headphones it may seem more natural to use another frequency band. Whichever band is chosen as the reference band always gets a score of 10. The other frequency bands are scored relative to the reference frequency band.
- If a frequency band sounds perfect for all the test tracks it gets a 10. If it sounds off, over- or under-emphasized on any of those tracks, it gets the lowest of the possible scores for all the tracks as defined below. For example, if the "highs" sound slightly emphasized on two tracks, and emphasized on another track, the emphasized score of 4 would be given for the "highs" frequency range for that headphone.
- Highly emphasized: 0
- Emphasized: 4
- Slightly emphasized: 8
- Just right: 10
- Laid back: 9
- Weak: 7
- Very weak: 3
- Formula applied: FR = (FR.deep.bass + FR.lows + FR.mids + FR.upper.mids + FR.highs) / 5, rounded to the nearest 1/10th.
- Overall Weighting: x4
Overall Listening Experience:
The Overall Listening Experience is a result of impressions during all the previous listening tests plus those from any other desired listening time.
- No test tracks are specified. Use impressions from all previous listening tests plus from listening to any other tracks you choose.
- Scoring
- An integer score between 0 and 10 is assigned.
- 10: Delightful. "I got chills, several times."
- 9: Very good. "They are really nice."
- 7 - 8: Good. "I like them."
- 4 - 6: Fair. "They are OK, nothing special."
- 0 - 3: Poor. "Forget about it."
- Formula applied: OLE = n
- Overall Weighting: x4
Comfort:
Pay attention both to initial impressions and to impressions formed after an hour or more. How do they feel on/around the ears, over the top of the head? If there is ear contact, how much pressure is there, how effective is the padding? Are the materials comfortable? Does the cable rub or cause discomfort? Do your ears get hot after awhile?
- Scoring
- An integer score between 0 and 10 is assigned.
- 10: Excellent. "I could wear these all day."
- 9: Very good. "Very comfortable, an occasional minor annoyance."
- 7 - 8: Good. "Comfortable, but there are limits."
- 4 - 6: Fair. "I can put up with them if I have to."
- 0 - 3: Poor. "Forget about it."
- Formula applied: CO = n
- Overall Weighting: x1
Design:
Appearance, build quality, durability, form follows function, usability and portability features, cabling, accessories, packaging.
- Scoring
- An integer score between 0 and 10 is assigned.
- 10: Excellent. "Wow, this is so cool in every way."
- 9: Very good. "Great design, only a minor annoyance or two."
- 7 - 8: Good. "Nicely done, but there are some flaws that are hard to overlook."
- 4 - 6: Fair. "OK, but some major problems."
- 0 - 3: Poor. "How could they get it so wrong?"
- Formula applied: DE = n
- Overall Weighting: x1
The Overall Performance Score:
And here is the Overall Performance Score. The Overall Performance Score is a weighted average, using the weightings defined for each category above:
OPS = (((2 * IM) + (2 * SS) + (2 * CL) + (2 * SP) + (4 * FR) + (4 * OLE) + CO + DE) / 18)
Other Factors - not part of the Overall Performance Score:
Several additional factors are considered which are not part of the overall scoring.
- $100 reference headphone: Yes or No. I have a dream of identifying a $100 reference headphone with extremely flat frequency response and high scores in the other categories.
- Drivable with portable media devices: Yes or No. Is this headphone sensitive enough to be driven to strong listening levels using a smart phone or portable listening device? (Tested at 88 to 90 dB volume level, one side, no obvious increase in distortion or sound degradation, with an iPad, an Android smartphone with no audio enhancements, and a budget laptop. The test device may be operating at maximum level. Tested with loud-recorded rock music, 8 to 12 dB dynamic range. A strong volume level, not super loud.)
- Usable without equalization: Yes or No. Is the frequency response good enough that I would be comfortable listening with it as is, without wanting to tweak it with available equalization?
- Isolation (if closed design): For closed headphones, how good is the isolation from outside sounds, and privacy going the other way? An integer score between 0 and 10 is assigned.
Headphone Roundup Scorecard
The Scorecard used to be right here. Now it is at the top of this document.
Note: This thread remains open for discussion about the Overview, the Scorecard, and the evaluation methods used. Please post comments about an individual headphone model or review in its own discussion thread.
In Summary
There are a lot of headphones out there, and a lot of headphone reviewers. The purpose of all this is to give you a different kind of tool to help you decide, based on these objective-subjective scores, which headphones you might like and want to buy. Yes that is an ambitious goal. Please let me know how well it works for you. Also remember, I gave this my best shot, based on my experience and my best judgments, but your own ears will be your own best judge. I welcome your constructive feedback, but alas, of course I cannot guarantee satisfaction based on the scores - short of joining you in a Venetian mind meld. That would cost extra. I do sincerely wish you the best in your headphone listening and purchasing decisions.
May your headphone listening sessions be many and awesome!
by Wayne Myers
AudiocRaver
Updates:
- 1-18-2013: Explanation in Imaging scoring method, definition of "Drivable with portable media devices."
- 2-1-2013: Explanation in Imaging scoring method, definition of "Drivable with portable media devices."
- 3-22-2013: New format for Scorecard; Scorecard now comes first in document; photos, frequency response curves, and frequency response demos added; links to Reviews now part of Scorecard.