Many times it will be Spotlight indexing the drive, but any app which accessed the drive could be holding onto it including the Finder. It is very annoying and happens way too often.
If you don't need Spotlight to index the drive, then you can tell Spotlight to ignore that drive, but using the directions in the following Apple article, but it won't immediately stop indexing although it will prevent Spotlight from interfering with ejecting the drive in the future:
Prevent Spotlight searches in files on Mac - Apple Support
If you do want/need the drive to be indexed, then you will need to identify the process using the drive and force quitting that process. You will need to use the Terminal command line to do this. Launch the Terminal app and begin typing the following making sure to leave at least one space after the final character:
sudo lsof
Then using the Finder, drag & drop the volume you want to eject onto the open Terminal window so that it auto fills the correct path to the volume. Press the "Return" key to execute the command. If the external drive volume is called "My_External_Drive", then the command will look something like:
sudo lsof /Volumes/My_External_Drive
You will be prompted for your admin user password. Nothing will appear on the screen as you type the password so you will need to press the "Return" key to submit the password.
If any processes are accessing that volume, then they will be displayed. You will need to use the Process ID (aka PID) for the following command in order to terminate that process. Start with the lowest numbered PID located in the second column of the output which will likely terminate many of the other processes that have a higher value PID. Use the following command template making sure to replace "<PID>" with the correct PID value shown in the output of the earlier command:
sudo kill <PID>
Here is an example:
sudo lsof /Volumes/M2_TM_Backup
Which gave me this output (I've only displayed a few lines since there are about 20 lines):
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mds 523 root 34r DIR 1,25 160 2 /Volumes/M2_TM_Backup
mds 523 root 35r DIR 1,25 3232 20 /Volumes/M2_TM_Backup/.Spotlight-V100/Store-V2/9B2B609F-B1FE-48BD-AFB7-05C9CE5C3848
mds 523 root 36r DIR 1,25 160 2 /Volumes/M2_TM_Backup
mds 523 root 37u REG 1,25 0 86 /Volumes/M2_TM_Backup/.Spotlight-V100/Store-V2/9B2B609F-B1FE-48BD-AFB7-05C9CE5C3848/journalExclusion
mds 523 root 38r DIR 1,25 160 2 /Volumes/M2_TM_Backup
mds_store 822 root txt REG 1,25 56 54 /Volumes/M2_TM_Backup/.Spotlight-V100/Store-V2/9B2B609F-B1FE-48BD-AFB7-05C9CE5C3848/dbStr-1.map.header
mds_store 822 root txt REG 1,25 56 58 /Volumes/M2_TM_Backup/.Spotlight-V100/Store-V2/9B2B609F-B1FE-48BD-AFB7-05C9CE5C3848/dbStr-2.map.header
mds_store 822 root txt REG 1,25 56 62 /Volumes/M2_TM_Backup/.Spotlight-V100/Store-V2/9B2B609F-B1FE-48BD-AFB7-05C9CE5C3848/dbStr-3.map.header
mds_store 822 root txt REG 1,25 56 66 /Volumes/M2_TM_Backup/.Spotlight-V100/Store-V2/9B2B609F-B1FE-48BD-AFB7-05C9CE5C3848/dbStr-4.map.header
You would then use the lowest PID value shown which in this case is 523 so the second command would be:
sudo kill 523
Run the "sudo lsof" command again (you can just press the up arrow key a couple of times to show the previously used commands to save you some typing....just press the "Return" key to execute the command displayed). If there are still processes using the drive, then use the "sudo kill" command with the next lowest PID value. Once no more processes are using the drive, then you should be able to eject the drive.