Had the starter contact problem, been there, I'm surprised you're not hearing a click, are you on a noisy street?
Once the engine warms up, it's usually ok. My problem was always with the first cold start of the day, so I wasn't to worried about getting stranded during the day. I left it untill it sometimes took 10 tries to get started, but this is bad for the starter
I changed the contacts myself, in decent weather under a covered drive, it;s not a side of the road job.
I did it with the starter in place, a very fiddly job, for the simple reason I didn't have the right socket set or curved spanner to get the sodding thing out.
The contacts were badly worn and I haven't has any bother with it since- 6 months