It's totally dependent on your source sample rate in your DAW.
Most developers utilize a variety of sample rates for their libraries natively. If you work at 96k, purchase those libraries at that rate.
There are a lot of libraries that you can purchase at your relative sample rate without needing to utilize Nebula's SRC internally. This helps with loading times and sound quality (but regardless, Neb's SRC is pretty darn good IMO).
Do not be dissuaded in purchasing lower sample rate based libraries such as AITB or some of CDSM's mastering or Producer's packs. They all sound amazing.
I understand your concern with this question, but this platform isn't based upon SRC or specific numbers. Its requirements for sound sculpting needs, because all pro libraries (as well as freebees) are all pretty much fantastic.
I'd start with a good tracking EQ like CDSM's N-ten84, a preamp collection like AlexB's preamps, rhythm minds tapes/filters and or CDSMs Ampex and tape booster (a tape and his tape booster are a huge requirement IMO). Depending on your musical styles I'd get a console to render your 'tracked' stems. They make a big difference.
Right now, I'm really into AlexB's Modern Logic console.
I'd start with a few libraries, namely with a tracking EQ, then maybe something like DocFear for a little bit more sculpting capabilities.
Feel free to ask any questions, as this is a fantastic and helpful community.