I swapped my GS for a 1999 S3 a couple of months back and I'm still loving every minute of it. It is fun and safe to drive, but it does make me drive much faster than my GS.
They're plenty fast enough as standard (there's little between the 210 and 225BHP models) but if you feel yourself needing a bit more they are really cheap to modify, with quite a few companies offering remaps to around 270BHP for around £500. I had a 5hr free demo of one such remap and it is awesome. I will be getting this done permanently quite soon. They handle well, but they do tend to understeer quite a bit when pressing on!
I went for an S3 because I wanted a change, and didn't fancy a Scooby (or at least I didn't fancy the image that goes with it). I do miss the space in the Lexus, but as I see it, I've got the rest of my life to buy a big luxury car.
As Colin says, the coil packs and diverter valves are a weak point. I replaced my diverter valve when I bought it - they're only about £15 so its not a big problem. Coil packs on the other hand are expensive. Mine are OK at the moment...
I get about the same fuel consumption out of my S3 as I did from my Lexus. I can get it as low as 20MPG, and upto about 33MPG depending on driving style. My normal mix of driving gives about 28MPG.