It does seem that, not only in these circumstances, but with some other programs, whilst they offer, during installation, to be the default, this still needs to be done manually, after installation.
However, It has been my practise, since Edge was first introduced, to change it from the default. It has never, since, troubled me. because of this, I would see no useful purpose in either disabling or, if possible, removing it. Particularly the latter, as it is also the PDF reader, which means I do not need to install another third party program.
Yes, it is running in the background. I have no explanation as to why, in the task manager. it is linked in to usage of IE only. But, it is not using any resources. If not using IE, I don't see any indication that it is running in the background. Again, it does not bother me.
As you may recall, there was a similar controversy, with a lawsuit, brought by the Europeans, regarding the automatic use of IE as the default in Windows 7. This resulted in the extreme measure of having a separate OS release for European countries. Maybe, eventually, the same thing will happen with Edge.