Non-running software doesn’t affect performance as it isn’t anywhere near your RAM or CPU. What often people perceive of bloat is frequently software dependencies that are likely to be used over the course of the OS’s usage.
Often I have found bloat free setups end up taking hours of digging out dependencies on multiple occasions. Life is too short. I have things to build.
It does take space on the hard drive. Can easily remove desktop shortcuts. Telemetry on open source software doesn’t usually happen without consent and you can turn it off (Firefox for example).
This is more of a feeling type thing. If it makes you feel good though, go ahead.