Sequoia stop automatic connection to file sharing by Apple ID

It's taken me a lot of googling and experimenting to figure out what the problem is and I don't know how to turn it off.

We have a Mac Mini running Sequoia. It has file sharing set up to three folders. There are users and groups set up with appropriate permissions. The intent is a remote Mac will access it by SMB and use their login account and password to access the appropriate shares. The Mac Mini and the remote Macs all have the same Apple ID (for business purpose).

After a lot of headaches and time trying to figure out how to stop automatic login to File Sharing, it appears that any remote Mac that is logged in with the same Apple ID can automatically connect. Talk about security fail!

Does anyone know how to turn this feature off? I get that if the Apple IDs are logged in that, in theory, you should be able to access all your devices, but there should be a lot more control over this.

I don't want to log the computers out of the Apple ID because I want to maintain Find My Mac under one account.