Exact same problem on my '90 LS400. Local machanic says replace starter motor is only option as it's a 4 hour job to get out, so you want to be sure it's fixed.
However, as you have the exact same problem (and mine is only from cold also), if one of use were sure what the problem was, would it be worth trying to fix the motor? My guess is that it is just a lubrication issue. If it were half-an-hour to get it off, it could be fixed by now!
Having just spent a massive amount on suspension bushes, I'm a bit committed to sorting this!
Just noticed your post is a year ago.... Did you fix it?
Dave/.