OK, I've finally worked out what the problems are with missing contacts in the car. It looks like the car can't cope with having fields in your contacts with multiple lines (typically addresses, but might be notes as well). Android copes with this fine, but the car seems to ignore anything that is too complicated for it.
To solve it, I had to export all my contacts from Google Contacts into Excel on my computer, remove the line breaks, then copy them all back to Google Contacts.