I gave someone my debit card to open his Apple Music. Will it be possible to remove my debit card that is connected to his Apple ID from my iPhone

Unless you know that person's Apple ID and password so you can login to their account via icloud.com , no. And even if you did, they may have written down all of the card information and can not only put it back in, but also change their password to prevent you from removing it again.

Never, never, EVER, give someone else your credit/debit card info. No matter how much you trust them. Anything they purchase on your card (especially a debit card) is your responsibility.

No

This is probably headed for card cancellation and debit card re-issuance with your financial provider.

Best let the financial provider know soonest too, as that can end your risk of (future) financial loss related to this.

What’s already gone may well be gone.

A gift card would be a better approach for these cases, should this or similar arise anew.