I have no issues giving up the S4 if I could only keep one. Why ? because an RS is an RS is an RS. The RS4 never fails to curl my socks . It's far from the fastest Audi I've owned as I used to have a Stg3+ B5 S4 but the RS just does everything well.
It looks angry and has timeless hips. The B7 RS4 6 speed is the best tranny we've ever made. The B8.5 6 spd is not as smooth or as direct.
My S4 is my DD as I want to keep the miles from adding up on the RS. It's on it's 2nd carbon cleaning but for me it's a very small price to pay for the reward of ownership. It's a visceral car that will never happen again...the RS5 would have been perfect had they went with the 7 speed manual option that was considered, but alas it's purely flappy paddled. Shame really.
Don't get me wrong, the S4 is lovely. Nice power, MMI is fine but imho is rather pedestrian compared to the RS4. With the Sport Diff it does rotates great but the V6 is far from an iconic motor.
I think the Rs4 and my 911 C4S is as good a pairing as I've ever had. I'm lucky and blessed to have both in my garage.
As an aside...I came close to buying your old WB S4. Get an RS4 ...it's more durable.