Apple Music App gets stuck on Apple TV Output

Sometimes, if I use the apple tv remote on my iPhone to control playback, and then try to play music on my iphone, my phone will attempt to play the music through the Apple TV. Even worse than this is that sometimes it gets stuck and will not let me switch playback to iphone only. I'm about to get rid of both of my Apple TVs and replace them with Rokus if I can't figure this out.

In case you are not understanding here is the scenario:

• kid wants to watch a movie on apple tv so i use apple tv remote on my phone
• i want to listen to music in my office while i work
• music starts playing on apple tv via apple music on my phone
• won't let me change airplay destination to my office speaker or to my iphone

The only fix I have found so far is to restart my iphone. I even made a shortcut to do so quicker since Apple thinks they don't need a restart button :)