Homekit Hubs not responding.

About a week ago I noticed that ALL of my homekit hubs and accessories are "not responding" when my phone isn't on my wifi network. I cant control anything remotely anymore. Anyone else have this problem?

I've even been home and just turned wifi off on my phone, and can still control everything via voice controls through my homepods. So its got to be something on the phone side?