Home Theater Shack 2015 High-End Amplifier Evaluation Event Reporting and Discussion Thread

See post 177. There were virtually no measureable differences that amounted to anything in either frequency response or impulse response. I have spent hours trying to find something and it just is not there. I really expected to find something in the impulse response, but it was actually more similar than the frequency response measures.

The bottom line is that we were not able to reliably detect differences nor reliably identify differences in ABX comparisons. the things that we made notes that we heard that had any correllation at all across listeners and sessions are detailed in the first post. They were mostly about the amps that we heard in more than one comparison, which leads me to believe that with more focused listening on less amps that we might be able to find some diffenences that hold up. Frankly, we probably had too many amps and tried to do too much. The next time we probably won't have more than 2 or 3.

Sorry to disappoint, but it is what it is. I don't think that it is fair to post every comment we wrote down on every amp that is uncorrellated. I am still uneasy with posting as much as I did because there may have been only one comment that supported each, and that is far from reliable. We can conclude that what we THINK we hear is highly variable and suspect.

We will have a much better idea about how to approach it next time.

I wish the test provided some conclusion is all. A null result is why blind testing is often criticized. I do think that a proper test method with this many amps would take more listeners, more time and a better room setup. I care less about the listener impressions than seeing the hard data in graph form. In a blind test fatigue and the stress to answer "right" bias listener results. Are we testing the amps vs listeners or the test vs. human psycology.

Numbers and hard data are unquestionable. In future I would recommend finding a set of speakers known to give amps trouble at key frequencies (probably bass impedance/inductance swings) and measure output vs frequency over a specific bandwidth. Then vary the output voltages higher and run the test over and over. Some amps will simply not "wake up" speakers until a certain voltage is reached. I recommended bookshelves because bass is often not their strongsuit and poor amps can result in thin sounding speaker response.

Remove the human element and the test becomes faster and conclusive. Then we can try to find out what the measurements mean in a listening experience.

Serenity Now said:

I wish the test provided some conclusion is all. A null result is why blind testing is often criticized. I do think that a proper test method with this many amps would take more listeners, more time and a better room setup. I care less about the listener impressions than seeing the hard data in graph form. In a blind test fatigue and the stress to answer "right" bias listener results. Are we testing the amps vs listeners or the test vs. human psycology. Numbers and hard data are unquestionable. In future I would recommend finding a set of speakers known to give amps trouble at key frequencies (probably bass impedance/inductance swings) and measure output vs frequency over a specific bandwidth. Then vary the output voltages higher and run the test over and over. Some amps will simply not "wake up" speakers until a certain voltage is reached. I recommended bookshelves because bass is often not their strongsuit and poor amps can result in thin sounding speaker response. Remove the human element and the test becomes faster and conclusive. Then we can try to find out what the measurements mean in a listening experience.

Click to expand...

No matter the initial conditions, variables introduced or placebos controlled, someone will always be dissatisfied with test results. That's the nature of testing. Hard data is not always the absolute arbiter of conclusion. Questionable recording practices and post-manipulation can come into play. I seriously doubt any of our panel engaged in such integrity-robbing practices. Rather, I believe they conducted themselves with the highest professionalism and exercised due diligence in set-up. Room acoustics and speaker positioning were already dialed in to the nth degree before the trials began. And why use hard-to-drive, specialty speakers unless their ownership proliferated throughout the mass market?

On one hand, you ask for hard data, but on the other you speak of "waking up" speakers; Where's the hard data for that? Remove the human element, and you have (drum roll, please) The Terminator Syndrome: machines measuring machines producing physical phenomena for other machines. The results can hardly be soothing. Right, Ahhnolt?

Sent from my iPad using HTShack

Whatever the shortcomings of blind listening tests are I have yet to see any other method proposed that is better.
If the proposal is to let measurements be the end of the argument then the electrical measurements on the many and various amplifiers that have been published in HiFi magazines should (if you understand basic electronics and orders of magnitude) lead to the conclusion that properly functioning amplifiers that are not overdriven will sound so similar that they are unidentifiable from each other in listening tests.

Every time any group starts an amplifier "shootout" there seems to be a groundswell of hope that there will finally be a conclusion that's different from all of the other controlled amplifier listening efforts that have come before.
But alas, if the levels are carefully matched and the listeners do not know which machine is powering the speakers the machines all sound the same.

I have been meaning to post my own observations and conclusions from the event. The posts over the last few days have prompted me to go ahead and get that done.

Thanks Leonard for being willing in take the time to dig through all of the data the way that he did. He has far more patience at that then I, and no one could have done a better job.

I had hoped that we would have much clearer results than we did, either that there were clear differences that we could prove, or that there were none we could hear at all. Instead we ended up with some of us able to hear some differences some of the time and only a little data to prove it. Concerning data which can be said to support any consistency of findings across the whole listening panel, Leonard has found what was there to be found and reported it already.

I will go ahead and post my individual observations for what they are worth, but only to be taken with a huge grain of salt, because they are impressions and that is all. If the others wish to post their impressions, they are welcome to do so.

First of all let me describe my evaluation process. I personally feel this is quite important because different people seem to have different ways of going about this. And if one has a listening style that works for him, that should probably be taken into account in the design of the blind testing that person will engage in. In other words, I might be able to set up a valid blind test method that would work great for me and throw Joe or Dennis or Leonard completely off, while there may be an approach that would work perfectly for some of them and leave me flat.

And this is one of the great difficulties in setting up tests like this. Someone with a good background in audio and acoustics and psychoacoustics and testing sits down and figures out a really good double-blind test method for ABX testing and 5 people walk into the room and it happens to fit the listening approaches of only one of them and he does well but the other four fail miserably, and the test overall shows no statistically significant data supporting the ability to tell a difference. Had the test been set up another way it might give a different result.

The ABX testing that we did made it a necessity that the evaluators rely upon extremely fine details being held in auditory memory for 30 seconds to a minute to be used in an AB comparison. Dennis appeared to do very well with this, while the rest of us did not. For me that was extremely difficult, as the fine differences that we were hearing were simply not something that I could capture in memory and carry forward in that way into a comparison 30 seconds to a minute later. Maybe with practice I could learn to do so but at the event I was not able to.

Here is what worked well for me. I felt fairly confident about the differences that I was hearing between amplifiers in sighted testing we did on the first day. The two amplifiers were set up, their levels were matched, we knew which was which, and we held the a/b switch in our hands while we listen to our own selected listening tracks. As I listened through my tracks I switched back and forth freely between the two amplifiers. Over time I started to recognize that there were certain passages of each track that seemed more likely than others to help reveal differences between the amplifiers, so I focused more on those parts of the tracks, but I also listened to other passages just in case something new would pop up.

When I heard a difference I tried to make a note of what part of what track it was that I heard it on, and what I heard, and where I felt I had time I would go back and repeat that to be sure that the difference was distinctive and easily identifiable. Remember these judgments were not absolute in any way but extremely comparative in nature, as will be seen in my impressions of some of the amplifiers the follow. By switching back and forth during those critical passages, I felt the contrast almost jumped out sometimes when the switching was done at just the right moment. Given the ability to do that repeatedly with a pair of amplifiers, I got to the point where I was pretty confident I could identify the difference consistently.

So if I was to set up a blind comparison around that listening style and try to get statistical data to show I could do it consistently here's how I would go about it. I would start out with the pair sighted so I knew which was which and go about the test as I have described and identify the characteristics comparatively between the two amplifiers. Then I would leave the room, have the test setter upper flip a coin and decide whether or not to swap the two amplifiers. When I came back into the room I would know it was the same two amplifiers but would not know if they had been switched or not. So my task would be to sit down and listen, switching back and forth and try to come to the same conclusion as I did before comparing with the same tracks and identify which of the amplifiers was A and which was b.

When done, I would leave the room and we would do the whole thing again, maybe 10 times in a row in a day. At the end of the day if with this process I was able to identify the amplifiers correctly say nine times out of ten, that would be a significant result.

On another day, the same process could be followed with another pair of amps. You can see that this could turn into quite a long, drawn out process with multiple people and multiple amplifiers. You can also see that someone else might try the same method and have it absolutely not work for him at all. And it would become difficult to find a way to work with a listening preferences of each listener and still end up with what one can call statistically valid results because it almost ended up being like different kinds of tests for each listener.

That is a problem that I see with throwing around broad statements like, can you prove it in a double-blind study? Which double-blind study? Who sets it up? What are the conditions?

Some will say that it is wrong to tailor the test to the listener, that it invalidates the study right off the bat. and again I would say that it depends on how you define what you were trying to accomplish. For the kind of differences we are talking about, I will go out on a limb and predict that if the only way it is approached is by trying to come up with a single generic test that has to fit all listeners with their different critical listening styles, then the testing is bound to show that those differences cannot be heard consistently across a broad listening audience under that kind of test.

But if somebody gets their gumption together to define an approach to accommodate individual listening style and crunch the numbers together at the end, and include the information about what those styles were, then we may someday end up with a real in depth test that shows that those differences can be discerned consistently. This could even be done in a way which accommodates those who prefer long term listening tests, as some say that that is the only way to really hear some of the fine differences. That has not been my experience so far, but it is very close minded of me to assume that it cannot work for someone else, or even to assume that would not work for me if I really give it a proper chance over time.

I would like to note one observation that I find somewhat humorous. In the ABX testing, my success rate at identifying the X amplifier was the worst of the whole bunch of us. I was wrong six out of seven times. In a way, that result is the most statistically significant of all the listeners at the event, I just had a mental flip-flop of some kind going on that led me to the wrong answer almost every time.


My Observations

These differences are comparative in nature, and almost impossibly small. I would never expect to be able to walk into a room and hear one of these amplifiers playing and say, "Hey, I recognize that particular sound as being the Parasound amp,” or the Krell amp or any other particular amp. And with my experience at this so far, I would be suspect of anyone who claims that they could.


Day 1, Sighted Pairings:

1 - Krell vs Parasound:
Krell, bigger sound
Parasound, not as big

2 - Denon vs Mark Levinson
Denon, brighter
Mark Levinson, rolled-off high end

3 - Emotiva vs Pass Labs:
Emotiva, slightly bigger bass
Pass labs, tighter
This was a fun pairing, I liked both amps.

4 - Van Alstyne vs Wyred4Sound, no difference noted

5 - Behringer vs Sunfire
Behringer, les bass
Sunfire, more bass

6 - Exposure vs Krell, no difference noted


Day 2, Blind Pairings:

1 - Denon vs Behringer
Denon, crisp highs
Behringer, silky highs
I preferred the Denon.

2 - Denon vs Mark Levinson
Denon, bass not as clear
Mark Levinson, bass seemed tighter, clearer
I missed the rolled off high frequencies of the Mark Levinson, which I heard in sighted testing.

3 - Exposure vs Parasound, no difference noted

4 - Wired for sound vs Emotiva
Wired for sound, solid highs, lively dynamics, richest string tones, punchy bass
Emotiva, punchy bass
I preferred the Wyred4Sound

5 - Wyred4Sound vs Sunfire
Wyred4Sound, a little shrill
Sunfire, alive, nice highs
I preferred the Sunfire

6 - Denon vs Pass Labs
I noted no difference between these two amplifiers, but my comments were that they were both very even, accurate, transparent, and natural, and that I'd like either of them.

7 - Denon vs Van Alstyne
Denon, okay, not quite as clear, a normal amp sound
Van Alstyne, super clear and detailed, space around all the sounds
I preferred the van Alstine


Future Work:

How about removing the room from the equation? Use the same setup, but at the speaker terminals attach an attenuator pad and buffer amp with leads to a different room, feeding a class A headphone amp and low-distortion headphones. With the right headphones, I can readily hear differences between headphone DAC/AMP models I am reviewing. Just an idea.


Conclusions:

The main takeaway here is that the differences are incredibly small, difficult to hear, and difficult to test for in a provable way. I would have probably been happy with any of these amplifiers if I had to walked into a room and heard it all by itself. I doubt I would have been able to say that any one of them was better or worse than any other under normal listening circumstances.

Wayne and I are of one mind on this. I was not much better than he was at matching the amps to what I thought I heard. I found myself asking the question, was it A or B that was more like X. I was taking notes but would still get confused. I have terrible short term memory for random data and have to repeat things to myself to be able to recall at all, and the short time was a problem for me. I would focus on what I was hearing and feeling and did not have time to let myself really get into the music. I was more successful when I just listened for feel rather than specific characteristics to compare the amps. The ones that I got right were purely on what the sound felt like to me, with no specific notes.

I have a bit of experience in testing and in behavioral research with multiple trials. One thing I have learned that was confirmed here is that subjects have to get comfortable with the testing context before you can get reliable results. I can see why so many people get hysterical at the idea of AB or ABX testing. It is very different than the way we normally listen.

I really believe that there were differences at times, but VERY small. It will take much more time and focus on just a couple of amps to tease them out consistently if it is possible at all. Looking back on it, trying to compare so many amps was a fools errand, even if it was tremendous fun.

I'll pull out my notes from the first day and post them. My impressions of the amps were sighted, but I can be sure that they apply to the right one. I got mixed up too much on the ABX comparisons to be sure that the comments would be about the right amp. The could easily be backwards.