I can kind of see where you are coming from, having owned an Audi previously and having a wife who works for Audi and has their company cars. Of all the German brands I think Audi interior quality and fit and finish is the best. However, we have a brand new 2.0T A4 S-Line with full leather interior and when it arrived I was fully expecting it to taint my feelings towards my year old IS250. Actually, the opposite is true, and every time I go out I find myself reaching for the Lexus' keys.
Yes, the A4 feels solid and the interior plastics are a bit better, but I find the keyless entry, the smooth V6 with Auto, the soft leather, heated seats of the IS just far more satisfying. The Lexus also rides better, handles better, has a better stereo, and feels equally solid on the road. There are certain things on the Audi that make you think "hey this is a well engineered car" - the door hinges for example, but then again you look at the fit and finish of the Lexus exterior and the technical specs of the engines and you see that they are actually superior to the Audi. I think Audi have a done a tremendous job over the last 20 years distancing themselves from VW while using a box of the same parts. It has certainly helped that they have "halo" cars like the RS4s and S3,4,8s, quattros etc, and I really think that Lexus need to look at this side of the business a bit more. Having said which, Audi have admitted that the headlong rush for sales has wrecked their customer service scores and look enviously at Lexus with their low volume, high satisfaction ratings.