Blind testing

Blind tests are absolute#

A blind test takes away everything except the sound. You don't know which one is the expensive one, which one has the vintage interface or which one came free with your DAW, so you have to make a decision with your ears alone. In an ABX test you hear A, you hear B, and then you hear X and say which one it is. Passing means you can reliably pick it. Saying "I can hear the difference" doesn't count, because perception is easily fooled.

This works both ways. Some real differences are hard to pass. A 1.5 dB EQ boost at 400 Hz on one of my own recordings is a subtle difference. I think you need trained ears to hear it, and it was difficult, but I could ABX it reliably. That is the kind of difference that is actually there. And always match the loudness first, or the test is worthless before it starts.

Whoever says it sounds better has to prove it blind#

Spend your money on whatever you like. If you buy a luxury watch that costs more than a house, I'll congratulate you. But the people who can afford those watches don't claim they tell the time better than an iPhone, because they know they don't.

The moment you say your expensive thing sounds better, you have made a qualitative claim, and now you have to prove it in a blind test if I am to believe you.

When I make a claim, I put the files up so you can test it yourself.

Isolate every variable except the one you are testing#

Someone mixes a song on a Neve console one day, mixes it again in Reaper on another day, and the comments argue about which one sounds better. But that comparison has a dozen variables in it: proficiency on each, the references he was using, how tired his ears were, whether he'd just had a coffee, his mood, where he was sitting, and if it was months later, possibly more earwax.

I made two quick rough mixes of the same song, on the same day, with the same plugins, one after the other. The difference between them was way bigger than the difference between a $350 API emulation and the most basic EQ built into the operating system. Your mixing is maybe a 9 out of 10 on the makes-a-difference scale and the EQ is a 1 out of 10, so comparing separate mixes measures the human, not the gear.

Set the bar as low as possible#

A good test asks the listener to do as little as possible. In my DAC loopback test the question was just "how many times did I switch?" No value judgement, no "which is which". If you can't even hear the switch, the debate about which sounds better is over.

Telling two things apart and preferring one are different questions#

You might correctly pick out the analogue-modelled mix and still prefer the digital one. So an identification result is not a preference result, and the other way round. My analogue plugin poll had to be read with exactly that in mind.

Price doesn't predict preference either. In a blind poll of three microphones, the majority preferred a cheap condenser to an AKG C414. Most people also assumed the odd-sounding one must be the cheap one. It wasn't. It was a dynamic mic, and that colouration is what dynamics are for. Microphones do genuinely sound different. The point is that the price tag tells you nothing about which one people will like.

Read the statistics, not the pie chart#

A pie chart that leans one way can still be random chance. You need the sample size and the p-value. A result that is extremely significant means what it means, and arguing with it just shows you don't understand statistics. My DAC poll came out at around p = 0.000000001.

And credit to everyone who commits to an answer in these tests, right or wrong. Committing to a guess is how you find out what you can actually hear.