I agree that it does distract your attention, but less though on hands free. What I find is just as bad though is the sat nav. Whenever I am in London, I spend half the time looking down at the screen looking for the next turning or a shorter route to avoid traffic.
It is obviously better than a paper map, but I have found that since I have had the sat nav, I am ducking and diving much more whereas with a map, you have a good read of it when your stationary and stick to the plan.