Yosemite and OCSPD / CRL Security Revocation

I ran into this problem recently and it took me quite a while to work out the cause and find a fix for it. So hopefully I can spare someone else the time and effort.

Background:

I create OS base images using AutoDMG. In this case I have created a base image using OS X 10.10.1.

I restore these base images to machines using DeployStudio. As part of that imaging workflow I also install some firstboot packages such as: munkitools, createuserpkg’s to create a couple of local admin accounts. I also run a firstboot script that triggers a munki run to install any software that the machine should receive.

Issue:

What I was experiencing was that during the munki run, installation of some software, most notably Microsoft Office updates would stall – stuck at the preparing stage. Even leaving this process for hours would not help the situation. In fact after about 2 hours, munki killed the process anyway.

To investigate further I SSH’d into the target machine while it was running the munkirun and watched what was happening in top. I noticed that ocspd was chewing around 100-150% CPU usage. Disk activity was minimal and the installer process was also stalled/stuck. It certainly wasn’t installing anything.

So what is ocspd?

So now I had a pretty good idea about why my installations and imaging was failing. It was something to do with ocspd. But what is ocspd and what does it have to do with installing software?

Well from the man page: 

ocspd performs caching and network fetching of Certificate Revocation Lists (CRLs) and Online Certificate Status Protocol (OCSP) responses. It is used by Security.framework during certificate verification.

Right. Now we are getting somewhere. So ocspd is being invoked by the security framework which is being invoked by installer because installer is trying to verify that the certificates that are used in the installer packages are valid and have not been revoked.

Ok now its starting to makes sense. What I also need to add in here is that these machines are on a school network. A school network that only provides internet access to the machines via an authenticated proxy. So when these machines are imaged and deployed they do not have access to the internet, this is only available once a student logs in and provides credentials to applications that request it via a GUI pop up.

So at deployment/imaging time ocspd is unable to contact the OCSP and CRL servers.

How long has this been an issue?

It would seem that this has been an issue since new installs of 10.7.5 which is when Apple set the OS to check for certificate revocation by default.

This setting is found in Keychain Access -> Preferences -> Certificates

I can only assume that I have not run into this issue before because previously I had been working with corporate or private education environments that do not have such a tight policy around internet access and the machines have been able to access the internet, or at least, through some whitelisting of URL’s the machines have been able to get access to the CRL and OCSP servers.

If you are interested in what URL’s ocspd is trying to connect to, a bit of digging has led me to believe that these are URL’s you would need to whitelist or allow unauthenticated access to. This may not be a complete list though and might be updated or changed with updates to the OS but on my 10.10.1 machine as of December 2014 this is what I was seeing:

crl3.digicert.com
crl4.digicert.com
crl.geotrust.com
crl.entrust.net
crl.verisign.com
ocsp.verisign.com
crl.apple.com
ocsp.apple.com
ocsp.entrust.net

The fix?

So we have two options, give the machine internet access so that ocspd can contact the ocsp and crl servers during installation tasks or disable the requirement to contact ocsp and crl servers. Option 1 is not possible for us, so I needed a way to change these settings from Best attempt to Off

After a lot of digging I was able to locate the required keys and set them via defaults as part of a first boot script that occurs before anything else in my imaging workflow.

The contents of my first boot script that achieves this looks like this:

#!/bin/bash

######################################################################################
#                                                                                    #
# Author:  Calum Hunter                                                              #
# Date:    17/12/2014                                                                #
# Version: 0.1                                                                       #
# Purpose: Resolve the issue of ocspd preventing installations to occur when client  #
#          is unable to connect to the internet to retrieve ocsp and crl information #
######################################################################################

echo "Disabling OCSP and CRL in /Library/Preferences"
defaults write /Library/Preferences/com.apple.security.revocation.plist CRLStyle None
defaults write /Library/Preferences/com.apple.security.revocation.plist OCSPStyle None

echo "Disabling OCSP and CRL in root's ~/Library/Preferences"
defaults write com.apple.security.revocation.plist CRLStyle None
defaults write com.apple.security.revocation.plist OCSPStyle None

exit 0

Props

Notable shoutout to William Smith (318 Inc) aka Talkingmoose on Jamf nation. A lot of the solution I have above is based upon a post he made on jamf nation about a similar issue someone else was having. Thanks buddy!

Advertisements

5 comments

  1. I have seen this solution to the oscp ‘problem’ in a number of posts around the net, but no one says 1) why apple decided to make this choice, and turn it on by default, and hide it in a place that only .0001% of users can find. and
    2) no one states what are the security implications of changing the settings.

    Would be good to understand that full context please.

    Like

    1. 1. apple does what they want when they want. they might document it if they feel like it. Don’t try to understand the rationale it will just frustrate you.

      2. It means that anything involving security certificates will now trust whatever is in the keychain. Thus if you have a root cert that signed another cert, then the system will allow that cert because you have a root cert that is trusted and it’s saying its all good. Rather than consulting a external database the CRL to determine if that root cert is valid and thus if the trust chain is valid. Ofcourse OCSPD and CRL are a more secure option. But for lab deployments and if you are managing your machines you are probably managing their configurations so its not an issue. Hope that helps a bit

      Like

  2. Greetings. I copied and pasted the above code into the terminal. The Preview app stop working, so I shut down the computer, then pushed the power button. The computer failed to boot. To solve the issue, I turned off the unbootable Mac and hooked up the unbootable Mac to a working Mac via fire wire. The working Mac was already on. I held down the “t” key while turning on the unbootable Mac. An icon for the unbootable Mac showed up in my Finder window on the working Mac. I went into the library of the unbootable Mac and deleted the whole file:

    /Library/Preferences/com.apple.security.revocation.plist

    by holding down command and delete. The unbootable Mac was then able to reboot after I shut it down and pressed the power button again. I hope this saves people some time. I suggest not using the code above.

    Like

    1. Then you are doing something wrong. I managed thousands of machines across thousands of sites and have no come across an error like this.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s