How to actually delete Rigid Bodies in Blender

I've been simulating ball physics in Blender - and it has been a little bit of a struggle. Yesterday while testing out different behavior of balls I realized that if I deleted some of the balls - Blender didn't actually remove their colliders. I couldn't see the balls themselves but I noticed that the deleted balls collided with the visible balls. I tried to restart Blender and clear the cache, but nothing worked. This morning I decided to create a simple test scene to learn what is actually going on. Is it a bug, or am I just not understanding how Blender's physics system is working?

This is the test scene in Blender: 

And this is how the Scene collection hierarchy looks like:

The cube is a passive rigid body and the spheres are active rigid bodies. To see which objects are part of the rigid body simulation, you can click on Display Mode and then on Blender File:

And then you navigate to the RigidBodyWorld collection. You can see that all five of our rigid bodies are in the collection. 

Now lets say you no longer want to simulate the balls in the "Balls 2" collection, which is just the blue ball: Sphere.003. To delete it, you select the "Balls 2" collection and click on "Delete Hierarchy" - and the blue ball is gone! 

But if you now press play you will realize to your horrors how the red balls are still colliding with an invisible blue ball! If you look at the RigidBodyWorld collection you will see that Sphere.003 is still in it, just grayed out. 

You try to restart, it doesn't work. You try to clear cache, it doesn't work. You google, you find nothing of value. The documentation only talks about adding to the collection. The only thing that works is to delete it manually. This will long-term become a hassle if you have to scroll through RigidBodyWorld and delete rigid bodies manually if you like me simulate many balls in many different collections... 

What you should have done instead of selecting the "Balls 2" collection and clicking on "Delete Hierarchy," you should have deleted Sphere.003 by clicking on it in the "Balls 2" collection and select Delete. Now it will be gone from the RigidBodyWorld collection as well!  

Something else...

While experimenting I tried to remove the Rigid Body World by clicking on "Remove Rigid Body World" in the Scene window and then I restarted Blender to see if something of value happened. 

When I added the Rigid Body World again, I saw to my horrors how everything fell down, including the Cube which is a Passive collider. For some reason, if you remove the Rigid Body World, restart, Blender sets all Rigid Bodies to active, so you have to manually set them to passive again!