Unable to add IMAP mail account with same email as my Apple ID

I can't add an IMAP mailserver with the email registered to my iCloud account.

My email is a@b.com and that's how my icloud is configured. I can't add an IMAP mailserver to my Mac with the email a@b.com (unable to verify username or password). However I can add otheremail@b.com no problem.

iCloud is not set up to manage my mail, and I don't want iCloud to manage my mail. I'd like to use Apple Mail to browse it but I can't add the account.