Working around failed Apple software updates with Munki

For at least a year MacAdmins have been dealing with Apple security updates failing if they are not installed soon after being downloaded. This is especially apparent to Munki admins who have Managed Software Center (MSC) configured to install Apple updates. Munki will invoke softwareupdate to download updates as soon as they are detected but users can defer the installation via MSC indefinitely by default.

If enough time has passed when the user finally decides to allow the logout / reboot for the update it will silently fail. The computer will reboot and the user will find themselves at the login window thinking the update completed successfully. However an hour or so later softwareupdate will again detect the update, download it and MSC will prompt the user to logout and install the same update they think they had just installed.

This is not a great user experience and has led to some frustration here at the newspaper. I’ve written a script that works around this behavior, read on if you are interested in the details.

No one in the MacAdmins Slack has come up with a conclusive reason for why these updates fail, but they have generally been referred to as “stale.” The behavior can often, but not always, be reproduced by manipulating the Date & Time in macOS. Simply changing the date of macOS on the computer (or virtual machine) by a month between downloading and trying to install an Apple security update will frequently cause failures in my testing with both Sierra and High Sierra.

Apple’s logging is less than helpful when this occurs, simply reporting:

softwareupdated: SoftwareUpdate: Invalid or incomplete local product 041-31490
softwareupdated: Removing local product after found to be incomplete

This problem is reportedly fixed in 10.14 Mojave, but admins who still have 10.12 Sierra or 10.13 High Sierra deployed could benefit from a workaround since it seems Apple won’t be providing a solution.

When looking for a way to improve the user experience the most obvious option to me was to have Munki enable bootstrap mode anytime an Apple software update that requires a restart is installed. This way after a failed installation attempt of a “stale” update, Munki (and softwareupdate) would run again immediately at the login window, detect, download and install the update from a “fresh” copy before the user can log back in. This would preserve the single, albeit extended, interruption of the user’s work.

Munki already does this when a user upgrades to a new version of macOS via MSC so it seemed like a logical extension to me. When I made the suggestion in #munki, developer Greg Neagle suggested that I could achieve the desired result with no changes to Munki’s code by using a postflight script. Munki executes postflight at the end of the run, after attempting all installations.

So the goal was then to write a script that would detect these failed Apple updates and enable Munki bootstrap mode prior to the reboot. Thankfully Munki logs information about these failures we can use to detect this situation fairly easily.

WARNING: Apple update Security Update 2019-001, 041-31490 may have failed to install. No record of success or failure.

Apple Software Update install of Security Update 2019-001-10.13.6: FAILED for unknown reason

It took a little trial and error but I’ve come up with a script that gets the job done in all my subsequent testing.

I start by checking for the existence of warnings.log because the absence of the file means no failed updates happened and we can exit right away and move on.

If warnings.log is detected we grep it for the specific warning text that is logged whenever these security updates fail. If we don’t get a match we can exit and move on.

If we do get a match the script will set bootstrap mode ensuring that Munki will run again at the login screen right after reboot.

#!/bin/bash

if
	[ ! -f /Library/Managed\ Installs/Logs/warnings.log ]
	then
	/bin/echo "No warnings from this Munki run, presuming no failed Apple updates exist."
	else
		if
			/usr/bin/grep -E 'Apple.*update.*Security.*failed' /Library/Managed\ Installs/Logs/warnings.log
			then
			/bin/echo "WARNING: Possible failed Apple security update detected, enabling Munki bootstrap mode." | tee -a /Library/Managed\ Installs/Logs/warnings.log
			/usr/bin/touch /Users/Shared/.com.googlecode.munki.checkandinstallatstartup
			else
			/bin/echo "No failed updates detected."
		fi
fi

MunkiReport already utilizes a Munki postflight to iterate through scripts in the postflight.d directory so piggybacking on that behavior simplifies execution. I’ve packaged this script and deploy it as a managed install into: /usr/local/munki/postflight.d/  Permissions need to be root:wheel 755.

The primary concern for putting this into action is the possibility of creating a bootstrap loop. Meaning that if the Apple update continues to fail on the subsequent attempts, Munki in bootstrap mode will download and attempt the install over and over after each reboot with no end. This has not happened in my testing, the second installation attempt after a fresh download has always succeeded, so I’m comfortable deploying my script to the fleet.

Let me know what you think or if you have any suggestions on how this workaround can be improved.