If a passenger is in the car with you, they can see when you need to concentrate, and understand periods of silence. A mobile caller however cannot see you, and may not even be aware of the fact you are driving. They may regard a short period of silence as simply a loss of signal and repeat what they have just said. I would also say the speech clarity of some phones can be rather poor, making you concentrate harder on what the caller is saying.
Having said that, I must confess that I found answering my mobile or ending a call the most hazardous as it required me to take my hands off the wheel.