Hi Billy,
It sounds like you have two pairs of AirPods 1st Gen and only one will connect to your Apple TV audio at a time. We know how useful it is to have two Bluetooth devices connected to your Apple TV and have some resource to share with you.
Make sure the Apple TV is up to date: Update Apple TV software
If you connect the AirPods with an updated iPhone, it should update the firmware of those AirPods, as well: Update your iPhone, iPad, or iPod touch
To share the audio, try out: Share audio with AirPods or Beats headphones
If the user signed in to the Apple TV (with their Apple ID) it should automatically connect the associated AirPods. That might be what is happening. Try to update your devices and connect the AirPods to your iPhone (to allow it to update your AirPods firmware), then connect them with your Apple TV using that last resource above.
Should you continue to have issues, does this only happen with the AirPods associated with the Apple ID currently signed in to the Apple TV?
Wishing you the best, cheers!