One does not need access to your iCloud Keychain in order to find you as they can simply use the Find My app and track you from there.
Two ways this could've been potentially done:
- He illegally gain entry into your device through the passcode, send a request to his device to share your location. He will accept the invite from his device.
- He send a request to your device to share your location with him. He illegally gain entry to your device and accept the invite he sent.
Therefore, I believe your iCloud Keychain is still safe and untouched as that only serves as a repository for logins and passwords. With that being said, you may wish to check your Find My app and see who you're sharing your location with and revoke all access.
You may also wish to change your passcode to your device, and change it to a longer passcode (more than 4-digits) if you haven't already done so.
As an added measure, you could change your Apple ID password to something very complex (letters, symbols,) and enable two-factor authentication if not already enabled -- If you think your Apple ID has been compromised - Apple Support.