I've come from a Jaguar XE, which is closer in size to the ES. I appreciate that a Jag is not a Lexus, but there will be some parallels in my experience, I think, that might be of interest.
Here is a summary of my thoughts:
The UX is certainly shorter than the XE, and the XE I believe is slightly shorter than the ES. This did not affect space in the front or rear cabin. In fact, family have said the UX was more comfortable in the rear, as the Jag had the massive bulge for the rear axle, so the middle rear passenger always suffered. The UX rises literally only a couple of inches in the rear middle floor section.
The UX is a little taller than the Jag was, so it's easier getting in/out
Turning circle is better. Not sure if that will be the same moving from an ES, but if the wheelbase is bigger I'd expect so.
Boot space is about 1/3rd less, but UX benefits from not being a saloon, so can get taller items in there. I've decided to purchase roof bars/roof box, so that resolves the issue on the rare occasion that I need more space for a holiday
The ES rear seats don't go down - this was the main reason I didn't buy one!!! That to me was a show-stopper, as at least once a month I will need to transport something that needs a bit more than the boot alone would allow.
There's no point in me offering driving comparisons between the two, as I've never driven an ES, so couldn't tell you if the UX is better or worse.
Hope this helps in some way.