Like most tests, null tests are useful or useless depending on how and when you use them. They can be useful in demonstrating placebo effects for instance. Personally, a simple null test helped me realize how susceptible I am to placebo effects, regardless of my knowledge/understanding. It also helped me identify a Nebula related bug in Wavelab one time.
There is no point trying to nullify everything, or wasting time on a test which isn't appropriate. But going on a null-test "witch-hunt" is also a bit too generalized I think.
but if you "null" something it doesn't mean anything.... it's like using an hammer for checking if something is working or not... for example there are many plugins around with a wrong fft, and they damage your audio even in bypass mode. You could null them perfectly even with the original source, so apparently everything is working as expected, but it's not.... it's pure nonsense. In nebula libraries you'll find amazing presets which are pretty flat in the spectrum, pretty trasparent on the dynamic side and with a very little harmonic distortion. They are useful but you'll not find anything relevant using a null test.... I think you could null a complex source for pretty all the time.... believe me, this null test thingy is unuseful at all. And I write that even if nebula is maybe the only plugin around which is not nulling for most of the time... but it would be just marketing.
Fair points about Nebula. But I'll give you some personal examples. For years I managed to convince myself that wavelab and Nuendo exports sound different. It was driving me crazy until I simply tested it and realized the outputs null perfectly. It was a humbling experience and very useful one. Later I realized that I actually "hear" differently depending on the color of the software interface... pretty funny actually.
Another time I was listening to some Nebula presets using wavelab montage and realized that I wasn't hearing any difference between a mono & stereo version of the same preset (tested on stereo file). I tested it and both versions nulled perfectly with each-other. I wrote the developer, which in turn tested it using his daw and showed me that the results shouldn't null. Wavelab montage was simply not handling this specific stereo preset properly...
well, it's like an hammer... using an hammer you could discover interesting things... but if you don't find anything relevant it doesn't mean they are the same. Probably differences in many daw are caused by little computational errors... there could be a lot of issues. In a perfect world the daw should just sum numbers coming from tracks. Well, guess what? sometimes it doesn't happen. Maybe you'll never discover it using a null test, otherwise developers would use it for debugging, and it never happens. If you want to compare really 2 daws you should pick numbers, create a wave using those random number, than create 2 tracks, and render. Than you should read the sum of them @32 bits or 64 bits precision, exactly. This is a PROPER test. You'll discover also other things like offset issues and other evil troubles, which are the real enemies in the daw world. For example fx channels are not synchronized, or delay compensation doesn't work perfectly for a particular plugin in a particular host... or things are messed in realtime mode for somewhat reason while they work perfectly if you render... of strange clips applied wheh samples are outside -1...+1 range... there are countless issues and differences.... sometimes a plugin or a daw makes an fft and than an inverse fft again even in bypass mode... numbers are pretty the same but not exactly... or tracks are not perfectly syncrhonized sometimes for this or that reason... or panning rules are different or achieved in strange ways.... who knows? compare numbers, this is a test!!!!
ngarjuna wrote:No, null tests are not ridiculous. They serve an important and simple purpose: to determine if there are differences between two sources. That is occasionally extremely useful and important on plenty of occasions.
Yes, like the time I needed an hammer hit on the head to stop obsessing about daw's and go listen to somthing better then a placebo effect..
a null test is a subtraction. It does make a sense if you find exactly zero. Which is not -140dB or -80dB. It's zero. A mistake on fft for example can't be detected unless you find exactly something different from zero, and it could be a single sample somewhere, and it could be very quiet. Trouble with daws, they are adding a noisefloor sometimes, in order to get rid of denormals. So I suggest to check numbers, which is the best way (samples are numbers... if you compare numbers you can't go wrong...). So just to make clear: if it doesn't null, it means they are different for sure. if they null it depends on your precision scale. If you find exactly zero they are the same, otherwise...
Wavelab measures all the way down to -144dBFS. If 2 24bit files null to such levels (in multiple tests) then they are identical. The dynamic range of a 24bit is 144dB. There are no hidden elements beyond that.