Apple Watch randomly restarting and it takes forever

I have had my Apple Watch 5 for about a year. About once a month or so, it would go into this restart mode. I didn't pay much attention to it since it was a rare event.

Now, this happens at least 2-3 times a week. The watch is seemingly "restarting" (apple icon in the middle with the white tick marks in a circle on the outside).

However, it takes 2 or 3 hours to complete this "restart" process.

I have tried doing a full reset, unpairing and repairing, nothing seems to work. I have to just wait it out.

Anything I can do to get this to stop?