Yosemite and OCSPD / CRL Security Revocation

I ran into this problem recently and it took me quite a while to work out the cause and find a fix for it. So hopefully I can spare someone else the time and effort.

Background:

I create OS base images using AutoDMG. In this case I have created a base image using OS X 10.10.1.

I restore these base images to machines using DeployStudio. As part of that imaging workflow I also install some firstboot packages such as: munkitools, createuserpkg’s to create a couple of local admin accounts. I also run a firstboot script that triggers a munki run to install any software that the machine should receive.

Issue:

What I was experiencing was that during the munki run, installation of some software, most notably Microsoft Office updates would stall – stuck at the preparing stage. Even leaving this process for hours would not help the situation. In fact after about 2 hours, munki killed the process anyway.

To investigate further I SSH’d into the target machine while it was running the munkirun and watched what was happening in top. I noticed that ocspd was chewing around 100-150% CPU usage. Disk activity was minimal and the installer process was also stalled/stuck. It certainly wasn’t installing anything.

So what is ocspd?

So now I had a pretty good idea about why my installations and imaging was failing. It was something to do with ocspd. But what is ocspd and what does it have to do with installing software?

Well from the man page: 

ocspd performs caching and network fetching of Certificate Revocation Lists (CRLs) and Online Certificate Status Protocol (OCSP) responses. It is used by Security.framework during certificate verification.

Right. Now we are getting somewhere. So ocspd is being invoked by the security framework which is being invoked by installer because installer is trying to verify that the certificates that are used in the installer packages are valid and have not been revoked.

Ok now its starting to makes sense. What I also need to add in here is that these machines are on a school network. A school network that only provides internet access to the machines via an authenticated proxy. So when these machines are imaged and deployed they do not have access to the internet, this is only available once a student logs in and provides credentials to applications that request it via a GUI pop up.

So at deployment/imaging time ocspd is unable to contact the OCSP and CRL servers.

How long has this been an issue?

It would seem that this has been an issue since new installs of 10.7.5 which is when Apple set the OS to check for certificate revocation by default.

This setting is found in Keychain Access -> Preferences -> Certificates

I can only assume that I have not run into this issue before because previously I had been working with corporate or private education environments that do not have such a tight policy around internet access and the machines have been able to access the internet, or at least, through some whitelisting of URL’s the machines have been able to get access to the CRL and OCSP servers.

If you are interested in what URL’s ocspd is trying to connect to, a bit of digging has led me to believe that these are URL’s you would need to whitelist or allow unauthenticated access to. This may not be a complete list though and might be updated or changed with updates to the OS but on my 10.10.1 machine as of December 2014 this is what I was seeing:

crl3.digicert.com
crl4.digicert.com
crl.geotrust.com
crl.entrust.net
crl.verisign.com
ocsp.verisign.com
crl.apple.com
ocsp.apple.com
ocsp.entrust.net

The fix?

So we have two options, give the machine internet access so that ocspd can contact the ocsp and crl servers during installation tasks or disable the requirement to contact ocsp and crl servers. Option 1 is not possible for us, so I needed a way to change these settings from Best attempt to Off

After a lot of digging I was able to locate the required keys and set them via defaults as part of a first boot script that occurs before anything else in my imaging workflow.

The contents of my first boot script that achieves this looks like this:

#!/bin/bash

######################################################################################
#                                                                                    #
# Author:  Calum Hunter                                                              #
# Date:    17/12/2014                                                                #
# Version: 0.1                                                                       #
# Purpose: Resolve the issue of ocspd preventing installations to occur when client  #
#          is unable to connect to the internet to retrieve ocsp and crl information #
######################################################################################

echo "Disabling OCSP and CRL in /Library/Preferences"
defaults write /Library/Preferences/com.apple.security.revocation.plist CRLStyle None
defaults write /Library/Preferences/com.apple.security.revocation.plist OCSPStyle None

echo "Disabling OCSP and CRL in root's ~/Library/Preferences"
defaults write com.apple.security.revocation.plist CRLStyle None
defaults write com.apple.security.revocation.plist OCSPStyle None

exit 0

Props

Notable shoutout to William Smith (318 Inc) aka Talkingmoose on Jamf nation. A lot of the solution I have above is based upon a post he made on jamf nation about a similar issue someone else was having. Thanks buddy!