How can I speed up Photos app with external HDD library on Mac?

In that case please download and run Etrecheck.  The free version is sufficient. Be sure to give it Full Disk Access.

Copy the report as shown in this animated screenshot

and use the Additional Text button to paste the report in your reply.

Then we can examine the report and see if we can determine the cause of the problem.

ReindeR Rustema wrote: … Switching between different 'Collections' in the sidebar takes between 10 and 30 seconds for example.

Seriously? That's awful! This shouldn't be happening, and it isn't characteristic of using a Photos Library from an external drive. Do you have a Library on your internal drive to see if it works OK?

To narrow things down, try these steps:

• Restart the computer (of course)
• Run First Aid with the Disk Utility
• Re-start in Safe Mode. This bypasses certain potentially disruptive processes, and it often helps. Safe Mode is different for different computers, so see this: 

           Start up your Mac in safe mode - Apple Support 

• Rebuild your Photos Library--close Photos and re-launch by option-command-clicking the app icon.
• Use another library if you have one, or make a small test Library with a few pictures in it and see if the same thing happens. You can create a new Library by closing Photos and then option-clicking on the Photos icon.
• Make a new user and see if the same problem recurs with it

The idea is to find out if the problem is with the Library, with your account, or with caches and login items, and stuff like that. One of these steps may fix the problem, but their failure to work also helps figure out what's happening.

Let us know what happens…

Apple recommends at least 10% of the internal drive be free. I've seen problems with less than 140 GB free out of 1TB-- I mean pretty crazy specific stuff in multiple apps, which disappeared when I cleared another 60 GB. I've not heard of such a requirement on an external drive.

What do you mean by "good"?

What speed do you get with the HDD?

This setup, which is probably the slowest SSD configuration, gives me a read and write speed of 500 Mbs:

An SSD in its own case with a communication chip can triple or quadruple that speed. If you're considering an external SSD I recommend those from OWC (MacSales.com).

Have you installed and run any "cleaning", "optimizing", "speed-up", anti-virus or VPN apps on your Mac?

You should get Blackmagic Disk Speed Test, free from the App store. If the results are slow, then the problem may be with the cables connecting the drive. Some cables (are you using USB-c?) look great and work fine for charging, but there's quite a difference in data transfer rate among them. The SSD that I'm using right now shows speeds of about 7000 MB/s, and my Photos Library on it behaves the same as on my Internal drive.

Yes-- something crazy is going on in your user account. As OT suggested, this can often happen when we run "Cleaning" apps or other system-wide gadgets. You should look first at the Login Items (I just search for that in System Settings) and see what's turning on by itself and messing with your system. I have 0 things in Login Items, a perfect number.

Unless you're using a true VPN tunnel, such as between you and your employer's, school's or bank's servers, they provide false security from a privacy standpoint.  Read these articles:  Pubic VPN's are anything but public and Security Risks: The Dangers of Using Free VPNs (eccu.edu).  

Did you have the VPNs activated in the second account that you tried? If not then the VPN apps have a high probability of contributing to the problem. I recommend you uninstall all VPN apps and their supporting files from your original user account and test Photos on the external hard drive.