Photos not syncing on new Macbook

Yes, you need to click on "Use as System Library" on your new Mac. That should allow you to enable iCloud Photos on that library. (The fact that the option was grayed out on the old Mac means it was active and so could not be further enabled.)

What system is running on your 2011 MacBook? What version of Photos?

When did the photos on the old Mac start being pixelated and uneditable? When you first enabled iCloud Photos there? When you migrated Photos to the new Mac? Some other time?

You may have just discovered the breaking point for how long a gap iCloud Photos can bridge (Photos 3.0 to 10.0 with photos coming from an iPhone 15).

It might be dependent on other factors.

What iOS runs on your iPhone?

Are you shooting High Efficiency or Most Compatible? Other Camera settings? Not ProRAW, I hope.

Just to confirm my understanding: Are only new photos pixelated and uneditable on the old Mac, then?