Because many libraries are sampled at different sample rates, you can't always match them with your current sample rate. I rarely work in 96k, but when I do, I'll never be stopped from using, say, the Mammoth EQ or Fate compressor from AITB if I think the job calls for it.
Regardless of the sample rate conversion and working with a lesser rate, it still SMOKES using an algorithmic plugin instead. If you can, it's always best to stay within your current session's sample rate, but don't be afraid of using a library with a different sample rate.
I agree with both of the responses! I would RATHER keep sample rates the same, and I Barendse and I have chatted about this before, but I frequently use presets that are not in the same SR as my project, and still sound great. I am from the analog audio world, and to me this is like deciding whether to use a favorite piece of hardware even though it may be routed with a less than amazing cable, or it is over-due for maintenance.... I am not striving for some form of perfection. That seems silly since we use Nebula to emulate analog signal chains, which are fraught with "imperfections" anyway (those imperfections are the things we are striving for!).