well, the flywheel has some lifetime. The less you stress it, then it has longer lifetime. If you change with no noise, I am sure it makes no stress to flywheel. If you change with the noise, there is some stress. I am not able to estimate, how serious is such a stress.
I'd say - get used to change gears without noise and then don't care :)