Using Azure Blob Storage as a highly-available CDP and AIA location for your internal PKI

I inherited a Windows PKI setup that had the Root CA installed on a Windows Server 2008 R2 Domain Controller, with the root certificate signed with a SHA1 hash. That DC was in the process of being decommissioned, and I also wanted to move to a better PKI design.

I’d previously set up 2-tier Windows PKI infrastructures with offline Root CAs, so I knew that this was the route I was going to take again (note that this is for an SMB environment).

There are plenty of good guides on configuring a 2-tier Windows PKI. In my opinion the best of the crop at the time of writing is probably Timothy Gruber’s 7-part guide to deploying a PKI on Windows Server 2016.

I would, however, highly recommend reading up on the topic before blindly following a guide. PKI is a complex topic, and you want to make the correct decisions up-front to avoid issues later on. Some additional recommended reading:

There are many recommendations around where to publish/advertise the AIA and CDP. Some of these include:

  • In the default location – LDAP and locally via HTTP on the CA server
  • To an internally-hosted web server, and then reverse-proxy connections from the Internet
  • To an externally-hosted web server

I’d already used Azure Blob Storage to store some other small files, so I thought I’d have a go at seeing if it’s able to be used for AIA and CDP storage. As it turns out, it’s quite easy to do, and you don’t even need to mess around with double-escaping like you would need to if you hosted on IIS or an Azure Web App:

TLDR; The CA saves the CRL files to the default location of C:\Windows\System32\CertSrv\CertEnroll, and AzCopy then copies them up to an Azure Blob Storage account that’s configured with a custom domain of pki.yourdomain.com

Here are the requirements to get this all set up:

  1. CDP and AIA on Enterprise/issuing CA configured to save to the default C: location, and also advertise availability at http://pki.yourdomain.com
  2. AzCopy installed on the Enterprise CA
  3. Allow outbound HTTPS/443 from the Enterprise CA to Azure Blob Storage
  4. An Azure Storage Account with blob storage configured for HTTP access. I’d recommend at least Zone Redundant Storage for availability.
  5. A custom domain name for the above storage account
  6. A folder in the blob storage named ‘pki’ (not necessary, but you’ll need to adjust the script if you don’t use this folder)
  7. A SAS key with read/write/change access to blob storage only (don’t assign more access than necessary)
  8. A scheduled task running hourly as NETWORK SERVICE to call the below PowerShell script
  9. Ensure that NETWORK SERVICE has modify permissions to the log location (default is %ProgramData%\ScriptLogs\Invoke-UpdateAzureBlobPKIStorage.log)

You’ll need to manually copy your offline root CA certificate and CRL to the blob storage location. This script is designed to copy the much more frequent CRLs and Delta CRLs from your Enterprise CA to blob storage.

As it turns out, AzCopy is perfect for this because it supports the /XO parameter to only copy new files. That allows us to schedule the script to run hourly without incurring additional data transfer costs for files that already exist in the storage account.

I wrote a PowerShell script that does the following:

  1. Checks that AzCopy is installed
  2. Determines if the C:\Windows\System32\CertSrv\CertEnroll folder exists
  3. Copies only changed files with extension .CRL to to the blob storage account
  4. Logs successful and failed transfers to %ProgramData%\ScriptLogs\Invoke-UpdateAzureBlobPKIStorage.log

You can find the script on my Github repo here: https://github.com/dstreefkerk/PowerShell/blob/master/PKI/Invoke-UpdateAzureBlobPKIStorage.ps1

PKIView

Use pkiview.msc on a domain-joined machine to check the status of your CDP and AIA

SAS.png

Generating a SAS with least-privilege for AzCopy to use. Note that you’ll need to set Allowed Protocols to HTTPS and HTTP, not HTTPS only

ScriptLogArchive

The script’s archive log, showing the successful transfer of the CRL and Delta CRL

As always, use this at your own risk and your mileage may vary. Please drop me a comment below if you have any questions, feedback, or run into issues with the script.

Fix: Can’t install iManage FileSite 64-bit due to installer complaining about mismatched ‘bitness’

Testing FileSite 64-bit, I ran into an issue on my own PC. I had 64-bit Office 2016 installed, but the FileSite installer refused to continue and presented me with the following message:

Dialog box: iManage Work FileSite (x64) requires that your computer has matching bitness with all Microsoft Office producs as well as any other iManage Desktop clients, Aborting Installation...

In an attempt to locate the cause of the issue, I fired up the trusty Sysinternals Process Monitor, and set up a filter to capture activity from msiexec.exe. I then further refined that filter to capture only RegQueryValue operations, and re-ran the installer.

Sure enough, Process Monitor picked up some instances of the installer reading from the registry to determine the ‘bitness’ of Office and other iManage products. In my case, there was a lingering registry entry that led the installer to conclude that I still had the 32-bit version of FileSite installed:

A screenshot of Process Monitor, showing a registry key at HKLMSoftwareWOW6432NodeInterwovenWorksiteClientCommonInstallRootbitness with a value of "X86"

Because I didn’t have any iManage products installed at the time, it was safe for me to delete the entire HKLM\SOFTWARE\WOW6432Node\Interwoven reg key.

The installer then ran successfully after this. Thank goodness for Sysinternals by Mark Russinovich.

Reset the CSC (Offline Files) database

Over the years I’ve had many issues where a Windows client PC just won’t connect to a share using the FQDN, but can connect using the NetBios name. There have also been plenty of occasions where the opposite is true, too.

I had the issue again today on a freshly-built Windows 10 machine. Folder redirection wouldn’t apply because the user’s desktop folder was inaccessible.

Then I came across this post on the Spiceworks forums that mentioned resetting the CSC database. I tried that, and folder redirection now works perfectly!

If caching is enabled on the share in question, and the CSC database is knackered, you’ll run into this same problem.

If only I’d known about this earlier in my 15-year IT career, it could potentially have saved a lot of headaches.

Add the following reg key/value. then reboot the PC:

reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Csc\Parameters /v FormatDatabase /t REG_DWORD /d 1 /f

Retrieving voicemail files from a Mitel 3300 controller, and converting them to WAV

Today I had the displeasure of having to figure out how to retrieve and convert voicemail files from some old Mitel 3300 controllers.

Thanks to this forum post which pointed me in the right direction.

  1. Connect via FTP using a proper FTP client like FileZilla, to the 3300’s IP address
  2. Navigate to /vmail/d/vm/grp/<extension>
  3. Grab the relevant file. They’re all G.711 U-Law format RAW audio files
  4. Grab SoX – https://sourceforge.net/projects/sox/, and extract it somewhere
  5. Run the following command:
sox.exe --channels 1 --type raw --rate 8000 -e u-law -v 1 -D <inputfile> outputfile.wav

Quickly uninstall an MSI on multiple computers using WMI

Today I was working on reducing our vulnerability attack surface, and needed to remove Adobe Reader from our servers. It appears that it was installed as part of a VM image, but never maintained afterwards.

Long story short, rather than mess around with ConfigMgr baselines or Applications, I decided to go the direct route. To top it off, PowerShell remoting’s currently playing up. I ended up using WMI via the method that I outlined in my previous post.

Given an array of server names in $servers:

$servers | %{Invoke-WmiMethod -Class Win32_Process -Name Create -ArgumentList 'MsiExec.exe /x "{AC76BA86-7AD7-1033-7B44-AA1000000001}" /norestart /qn' -ComputerName $_}

Trigger a remote GPUpdate without PSRemoting or PSExec

I recently enabled Windows Firewall on an unused server via GPO, but forgot to include the inbound RDP exception. This, of course, kicked me off my RDP session.

Rather than wait ~90 minutes for my revised GPO to take effect, I found that I could trigger a GPUpdate remotely using WMI (WinRM wasn’t enabled, and I didn’t want to use PSExec)

The following command does the trick:

Invoke-WmiMethod -Class Win32_Process -Name Create -ArgumentList "gpupdate.exe" -ComputerName <computername>

Correct Horse Battery Staple for PowerShell, AKA Random Memorable Password Generator

I’m a fan of using correcthorsebatterystaple.net for generating temporary passwords for users, rather than always using a static password. The site itself is a reference to the XKCD webcomic, and yes, I’m aware that there are plenty of opinions on the web about this topic.

password_strength

I’ve had the idea in the back of my mind for a while to see if I could replicate the site’s functionality in PowerShell. I noticed that the source code for the site is on GitHub, so I ducked over there to check out the word list.

I found that it’s possible to replicate most of the functionality of the site with just two lines of PowerShell (although it doesn’t result in very readable code):

  1. I used Invoke-WebRequest to grab the word list from GitHub
  2. I then expanded out the Content property, and split it up given the comma delimiter
  3. I then used a combination of the Range operator, Foreach-Object, [string]::join,Get-Random and the TextInfo class to generate a given number of passwords along these rules:
    1. 4 random words, each with the first letter capitalised
    2. A separator in between
    3. A random number between 1 and 99 at the end

Note that this isn’t failure-proof, and isn’t intended to be used in any complex scenario. There’s no error handling, and not much flexibility built in. It’s just a quick function you could put into your PowerShell profile.

You can, at least, do the following:
A PowerShell window displaying the output of Get-RandomPassword. Also shows the function being called with a Count parameter as well as a separator parameter

Here’s the code: