APFS Encrypted: slow write speed on magnetic disks

Question

Level 1

25 points

APFS Encrypted: slow write speed on magnetic disks

I am using a QNAP TR-004 hardware RAID over USB 3.1 on a Mac Mini M1 (2020) running macOS Sonoma 14.1.1.

The 4x 8TB drives in the RAID array are arranged in a RAID 5 to present one logical 24TB drive. Each disk is the same model of enterprise class HGST data center drive (not SMR). They're rated at 205 MB/s for both read and write, and real world benchmarks back that up. Users of this hardware RAID enclosure report over 200 MB/s write speed in some cases. USB 3.1 is capable of more than that, so all pieces of hardware should support something in that range. The RAID array reports that SMART status of all 4 drives is good.

I have done something a little unusual that I now think might be giving me weird results: I formatted the logical 24TB drive as APFS encrypted. I now realize that APFS was designed for solid state drives and is not ideal for magnetic drives, nor necessarily for striping across 4 drives at once. Obviously encryption has some speed penalty, but I'm willing to pay that if it's reasonable.

But the performance I'm getting is not reasonable. I can read from the array pretty consistently above 160 MB/s, totally acceptable given the encryption. But I can only write to it at 30 MB/s, again very consistently.

I have tested with a 40 GB video file, copying it between the internal SSD (APFS), the RAID array, and a different magnetic drive (same manufacturer) formatted as HFS+. The internal SSD can read at 3 GB/s and write at 1 GB/s, and the HFS+ magnetic drive can write at 120 MB/s, both expected. Only the encrypted APFS magnetic array cannot top 30 MB/s write.

Is APFS encrypted performance really that bad on magnetic drives? Or with RAID 5 for some reason? I wish I could test the RAID enclosure with a different filesystem, but I don't have 4 other drives, and it's also kind of hard to do any write tests on each drive individually outside the array at this point. Any thoughts out there?

Mac mini

Posted on Nov 15, 2024 6:35 PM

Reply

Answer 1

Top-ranking reply

VikingOSX

Community+ 2024

Level 10

116,759 points

Nov 16, 2024 7:15 AM in response to cazoirarc264

You are far out of date with your Sonoma installation. The current version is v14.7.1. I recommend that you apply that update. Your Software Update panel will likely show Sequoia v15.1 and if you scroll downward, you will see a link named something like an additional update is available. If you click on that you will see Sonoma v14.7.1 and probably Safari 18.1 available for installation. When selected these, and not Sequoia will be updated.

The operating system and APFS are intended for solid-state drives that have orders of magnitude performance gains over rotational drives. Encryption/decryption will take a toll on performance. Especially on rotational drives.

You should check if your QNAP RAID has any outstanding firmware updates available for it and apply those. Is that USB 3.1 cable rated for Gen 2 (10 Mbps) performance or is it a Gen 1 cable (5 Mbps)?

Your Sonoma v14.1.1 is far too early for Apple to have applied any performance updates to it so v14.7.1 may make a difference despite your drives in use.

Even if you chose the 16 GB RAM option for your M1 mini, it may not be sufficient for your applications that are running and the QNAP overhead.

Reply

Answer 2

steve626

Level 6

11,501 points

Nov 16, 2024 1:17 PM in response to cazoirarc264

You should have a look at this technical article:

https://bombich.com/blog/2019/09/12/analysis-apfs-enumeration-performance-on-rotational-hard-drives

It is 5 years old but I believe it is still valid.

"After the very first simulation, APFS starts at a deficit — APFS takes three times as long to enumerate a million files on a rotational disk compared to HFS+ enumerating the exact same collection of files on the exact same hardware. This result on its own is staggering. As you add and remove files from the volume, however, the performance continues to decline. After just 20 cycles, APFS enumeration performance is 15-20 times worse than HFS+ performance."

I use a number of external mechanical drives as Time Machine and SuperDuper target backup drives, none (except one, see below) are set up as RAID. All of mine are APFS. I have never seen write speeds as low as 30 MB/s; when I have checked the speeds were always 80 MB/s to 100 MB/s. I also use some external SSDs for Time Machine and SuperDuper clones, and those achieve 500 MB/s to 1000 MB/s. The computer I measured these on is a 2019 MacBook Pro. I "measure" write speed simply by noting that, say, a 400 GB initial clone or initial Time Machine backup took about an hour from start to finish on a mechanical drive and about 10 minutes on the SSD. That said, these differences are not really noticed doing incremental backup updates because those are so quick, maybe less than a minute, or sometimes a few minutes.

But it sounds like you are doing serious production work not just incremental backups. Based on the article referenced above, you will see very significant slowdowns with APFS on mechanical drives for production work, and it will get worse over time.

I am wondering if you would be much better off using HFS+ for the mechanical drives in the manner you are using them.

I do have one RAID 0, an older Seagate drive that has been in use for more than 10 years. That is a pair of mechanical drives, HFS+, and it achieves 200-300 MB/s, much faster than what I have seen with individual mechanical drives on HFS+ or APFS.

I think the combination of APFS on mechanical drives plus the overhead of RAID and of encryption are combining to degrade your throughput.

Reply

Answer 3

cazoirarc264 Author

Level 1

25 points

Nov 17, 2024 12:34 PM in response to cazoirarc264

steve626 said:

You should have a look at this technical article:

https://bombich.com/blog/2019/09/12/analysis-apfs-enumeration-performance-on-rotational-hard-drives

It is 5 years old but I believe it is still valid.

Thanks, I've actually seen that one already, but it's not clear to me if it's directly applicable to this use case. The author is demonstrating very poor filesystem modification, basically creating many small files and then listing them, but it wasn't clear to me that would translate to creating one large file:

That article said:

When I refer to "filesystem performance," I'm specifically referring to how long it takes the filesystem to do transactional tasks. Read and write performance depends almost entirely on the speed of the media, so I wanted to factor that out of my tests. Enumerating the contents of the filesystem is a good exercise of filesystem transactional performance, so I decided to test the enumeration of 1 million files on each an APFS and HFS+ filesystem over a period of simulated modifications to the filesystem.

But it is indeed suggestive that APFS is not so good for mechanical drives in some ways. And yet your example seems to suggest that non-encrypted APFS should at least work much better than I'm getting:

steve626 said:

I use a number of external mechanical drives as Time Machine and SuperDuper target backup drives, none (except one, see below) are set up as RAID. All of mine are APFS. I have never seen write speeds as low as 30 MB/s; when I have checked the speeds were always 80 MB/s to 100 MB/s.

As for what I'm actually doing with them:

steve626 said:

But it sounds like you are doing serious production work not just incremental backups.

Indeed I'm using this for video/art production, but I'm not using it as a scratch disk or anything, it's more for network storage, and mechanical disks should be at their best for sequential read/write.

So I'm sort of still stuck on the factors that are a little hard to change without replacing all 4 four drives in the RAID and trying HFS+ or APFS without encryption:

APFS encryption might be an especially bad fit for mechanical drives for some reason
APFS (with encryption?) could be an especially bad fit for RAID 5 on mechanical drives for some reason
One of the drives could have some problem that isn't reported by SMART

I'm considering ruling out (3) by running some diagnostics on each disk by connecting it to a Linux machine, but that's a whole undertaking. I wonder if anyone else has ever tried encrypted APFS on mechanical disks in a RAID. I do wish I hadn't started with that at this point.

Reply

Answer 4

cazoirarc264 Author

Level 1

25 points

Nov 17, 2024 12:35 PM in response to cazoirarc264

Thanks for the suggestions. Some good ideas there. I just updated to Sonoma 14.7.1 but no change.

VikingOSX said:

You should check if your QNAP RAID has any outstanding firmware updates available for it and apply those. Is that USB 3.1 cable rated for Gen 2 (10 Mbps) performance or is it a Gen 1 cable (5 Mbps)?

I did already check for firmware updates, and also reached out to QNAP and they said it should be capable of much more than I'm getting. I'm using the USB cable that came with it, not sure what it's rated for, but I'm not getting even anywhere near 5Gbps as is, more like 240Mbps.

VikingOSX said:

Even if you chose the 16 GB RAM option for your M1 mini, it may not be sufficient for your applications that are running and the QNAP overhead.

The mini is really just a file server, not actually running applications. CPU and RAM are barely impacted during transfers, the machine is actually overspecced for the job of storage server. I checked SMB performance for other drives, and it is fine, maxing out the gigabit ethernet if the drive being written to supports it, and offering the same write performance to the HFS+ spinning drive as doing the copy locally. This was the use case that first had me scratching my head, so I replicated the slow write speed locally, ruling out SMB.

Reply

Answer 5

cazoirarc264 Author

Level 1

25 points

Nov 17, 2024 9:15 PM in response to Barney-15E

Barney-15E wrote:

Have you looked at System Information to make sure the drive is connected to a high-speed bus?

Perhaps it is connecting at USB2 speeds for some reason.

Yes, it's definitely connected to the USB 3.1 bus, which System Report shows as capable of 5Gbps, and it can read at higher than USB 2.0 speed, just not write.

Reply