That MacOS hasn't removed from OS-level /etc/ssl seems like a big deal, affecting more than just OS-supplied git. Is /etc/certs not used by most graphical Mac software, I guess? The chaotic state of OS-level cert management, on Linux as well, is pretty dismaying.
If Apple is still trying to brand themselves as security-forward, you'd think removing the revoked CA certs from /etc in a system update would be a pretty easy thing to do?
Following the eMudhra incident I made a tool to distrust all except the 30 most essential CAs on my Linux and Windows systems (https://github.com/hyperreality/distrust). I was shocked at how messy the situation is on Linux, with browsers, CLI tools, programming languages all having their own stores and distros building the bundles differently (update-ca-certificates in Debian vs update-ca-trust in Arch or Fedora). Or like Firefox not using the certs in /usr/share/ca-certificates/mozilla/ but its own per-profile NSS store instead. There has to be a better way.
Browsers were the original TLS trust store vendors. Operating systems started shipping trust stores because their OS came with a browser, not because they were the natural root of trust. Browsers also have tools to update revocation near-live, while operating systems lag behind by hours, or days in this case.
For your tool, remember to nuke Windows' automated certificate authority updates (separate from normal Windows updates) and possibly the automated intermediary cache. Microsoft publishes a list of trusted CAs/intermediaries periodically and they get automatically applied to Windows so websites don't need to send the full cert chain during TLS negotiation, and it has a tendency to cache trusted intermediaries as well.
Mozilla's trust store is one of the first to get updated when a revocation happens, as well as Google's CRLsets. Downloading full CRLs repeatedly would be a waste of time and bandwidth, that's why there's a specific distribution method for CA updates. I would extend your tool with a daemon that checks for revocation automatically, so you don't accidentally eMudhra yourself while browsers and the OS get updates next time something like this happens.
I was also surprised to find Java’s key tool returning successfully for the delete request without actually doing anything: list, delete, and list again would still show the CA. Debian had this right during the 90s, all they had to do was follow the system convention.
This feels like all of the C code getting attacked now where everyone spent the last three decades building up cruft because fixing it would be a chore, and now the separate bills are coming due all at the same time.
I used to manage OS level trust root management for all internal machines at a hyperscaler. Hundreds of thousands of machines. Even when you have an edict from a common VP or CEO, and most people are even using the same linux distributions, it's hard to get people to adhere to or even understand that there is a standard. Throw containers, embedded devices, windows and mac in and it's just a total nightmare unless you exert strict control on how every single machine is set up. Which, of course, is impossible because cowboys gonna cowboy.
Fundamentally, people don't want to think about the intricacies of how we decide whether or not to trust a particular cert. They want it to just work without thinking about it. This includes that if they have a custom private root they need to trust, they just drop it various places until it starts working and call it done. And sometimes they do weird things like adding intermediates to their trust store because they were trying to call a service that has their cert chain configured incorrectly, and that "fixes it". Again, it's a nightmare.
The fact that linux, a system notoriously cobbled together by 873 different factions on the internet, is even more of a mess is not surprising in the least.
Apple's official API had revoked the certificate within hours. Only the Unix compatibility layer they sort-of maintain has this problem. This is hardly the first or only outdated Unix compat tool they ship with their OS.
Not great for unaware developers, but most macOS users are fine. Those who are aware can follow the steps in https://support.apple.com/en-us/149555 until Apple bothers to roll out an update.
The thing that makes this especially odd is that Apple apparently has a fast-track way of shipping security patches for the OSs that don't require the user to go through the lengthier OS upgrade process. But for some reason, they haven't used it here.
That fast-track method was used, but it only applies to "graphical" software which really means, software that uses Apple's macOS specific APIs and frameworks, not the lower level UNIX stuff like /etc/ssl
I suspect the fast-track background security updates can only update the databases and revocation lists that live in the proprietary APIs, not any arbitrary file on the system
It’s not an excuse but an explanation: there are separate stacks owned by different teams and they haven’t been forced to collaborate. If you use an internal CA you’ve already had to deal with the easier additive version of this problem because that fails in an obvious, less harmful manner than quietly accepting something which needed to fail.
It doesn't explain why Apple who 1) sells itself on security and 2) ease of use doesn't 3) update what it installed and protect its users. Why are they not doing those things?
I wasn't trying to excuse anything, just explain that they haven't done nothing, but also why it's still an issue. I agree with you they've dropped the ball here. If I were a betting person I'd guess this is just something that slipped through the cracks and there will be changes whether that's to the release process for pushing OS updates, to the fast-track security update process, or to the OS so that everything is working off the same trust store
I don’t think it’s gross incompetence, since they documented the steps to remove. They may have some unexpected dependency on the file? Looks mysterious from the outside.
People would have to use it in some drastic catastrophic way for them to force it out in an update, which they might push it out in the next macOS update, which don't come nearly as frequently as Windows bloatware updates do.
This seems important enough to just send out a patch update that removes the cert and does nothing else. But apparently at least a couple people at Apple disagree; or, something I'm not familiar with makes that type of update impossible.
I mean, I agree with you, I'm just basing my thoughts of what I've seen managers at different companies prioritize as "important" even when engineers cry that a security issue should be of higher priority...
You have to admit that “just a quick patch” isn’t automatically low-risk. You could cause a large blast radius if you rush any patch out without proper testing.
I agree 100%, but I keep going back to the blast radius of a compromised CA and my gut instinct is still "get rid of it and push a hotfix" but I completely acknowledge it's a lot more complicated to ship a MacOS update than just branch off of master and push a new commit, and my experience is almost entirely the latter.
I think the main problem is that the weak eMudhra intermediates were cross-signed by Sectigo. Certificate revokations are needed to handle this correctly.
Dropping the Sectigo root along with the eMudhra root is the easy and safe solution, and maybe this is what Apple should have done (it’s the workaround they recommend!). But it means that some small percentage of certificates that are fine won’t be accepted.
> Is /etc/certs not used by most graphical Mac software, I guess?
Correct, those usually use Apple's networking APIs which all go through their security stack, which already picked up the OCSP revocation within about 2 hours in most cases.
this only affects terminal apps, which the vast majority of MacOS users don't run; the fix for graphical Mac software is already patched
my guess is their thinking is that if you're running those apps on Mac, you're a developer or know what you're doing, and can edit /etc/ssl/cert.pem until they push an update
they also have to consider that such users may have custom changes to their cert.pem file, and they probably don't want to rush an update that would potentially fu-bar those changes
If Apple is still trying to brand themselves as security-forward, you'd think removing the revoked CA certs from /etc in a system update would be a pretty easy thing to do?