Using Azure Blob Storage as a highly-available CDP and AIA location for your internal PKI

I inherited a Windows PKI setup that had the Root CA installed on a Windows Server 2008 R2 Domain Controller, with the root certificate signed with a SHA1 hash. That DC was in the process of being decommissioned, and I also wanted to move to a better PKI design.

I’d previously set up 2-tier Windows PKI infrastructures with offline Root CAs, so I knew that this was the route I was going to take again (note that this is for an SMB environment).

There are plenty of good guides on configuring a 2-tier Windows PKI. In my opinion the best of the crop at the time of writing is probably Timothy Gruber’s 7-part guide to deploying a PKI on Windows Server 2016.

I would, however, highly recommend reading up on the topic before blindly following a guide. PKI is a complex topic, and you want to make the correct decisions up-front to avoid issues later on. Some additional recommended reading:

There are many recommendations around where to publish/advertise the AIA and CDP. Some of these include:

  • In the default location – LDAP and locally via HTTP on the CA server
  • To an internally-hosted web server, and then reverse-proxy connections from the Internet
  • To an externally-hosted web server

I’d already used Azure Blob Storage to store some other small files, so I thought I’d have a go at seeing if it’s able to be used for AIA and CDP storage. As it turns out, it’s quite easy to do, and you don’t even need to mess around with double-escaping like you would need to if you hosted on IIS or an Azure Web App:

TLDR; The CA saves the CRL files to the default location of C:\Windows\System32\CertSrv\CertEnroll, and AzCopy then copies them up to an Azure Blob Storage account that’s configured with a custom domain of pki.yourdomain.com

Here are the requirements to get this all set up:

  1. CDP and AIA on Enterprise/issuing CA configured to save to the default C: location, and also advertise availability at http://pki.yourdomain.com
  2. AzCopy installed on the Enterprise CA
  3. Allow outbound HTTPS/443 from the Enterprise CA to Azure Blob Storage
  4. An Azure Storage Account with blob storage configured for HTTP access. I’d recommend at least Zone Redundant Storage for availability.
  5. A custom domain name for the above storage account
  6. A folder in the blob storage named ‘pki’ (not necessary, but you’ll need to adjust the script if you don’t use this folder)
  7. A SAS key with read/write/change access to blob storage only (don’t assign more access than necessary)
  8. A scheduled task running hourly as NETWORK SERVICE to call the below PowerShell script
  9. Ensure that NETWORK SERVICE has modify permissions to the log location (default is %ProgramData%\ScriptLogs\Invoke-UpdateAzureBlobPKIStorage.log)

You’ll need to manually copy your offline root CA certificate and CRL to the blob storage location. This script is designed to copy the much more frequent CRLs and Delta CRLs from your Enterprise CA to blob storage.

As it turns out, AzCopy is perfect for this because it supports the /XO parameter to only copy new files. That allows us to schedule the script to run hourly without incurring additional data transfer costs for files that already exist in the storage account.

I wrote a PowerShell script that does the following:

  1. Checks that AzCopy is installed
  2. Determines if the C:\Windows\System32\CertSrv\CertEnroll folder exists
  3. Copies only changed files with extension .CRL to to the blob storage account
  4. Logs successful and failed transfers to %ProgramData%\ScriptLogs\Invoke-UpdateAzureBlobPKIStorage.log

You can find the script on my Github repo here: https://github.com/dstreefkerk/PowerShell/blob/master/PKI/Invoke-UpdateAzureBlobPKIStorage.ps1

PKIView

Use pkiview.msc on a domain-joined machine to check the status of your CDP and AIA

SAS.png

Generating a SAS with least-privilege for AzCopy to use. Note that you’ll need to set Allowed Protocols to HTTPS and HTTP, not HTTPS only

ScriptLogArchive

The script’s archive log, showing the successful transfer of the CRL and Delta CRL

As always, use this at your own risk and your mileage may vary. Please drop me a comment below if you have any questions, feedback, or run into issues with the script.

Fix: Can’t install iManage FileSite 64-bit due to installer complaining about mismatched ‘bitness’

Testing FileSite 64-bit, I ran into an issue on my own PC. I had 64-bit Office 2016 installed, but the FileSite installer refused to continue and presented me with the following message:

Dialog box: iManage Work FileSite (x64) requires that your computer has matching bitness with all Microsoft Office producs as well as any other iManage Desktop clients, Aborting Installation...

In an attempt to locate the cause of the issue, I fired up the trusty Sysinternals Process Monitor, and set up a filter to capture activity from msiexec.exe. I then further refined that filter to capture only RegQueryValue operations, and re-ran the installer.

Sure enough, Process Monitor picked up some instances of the installer reading from the registry to determine the ‘bitness’ of Office and other iManage products. In my case, there was a lingering registry entry that led the installer to conclude that I still had the 32-bit version of FileSite installed:

A screenshot of Process Monitor, showing a registry key at HKLMSoftwareWOW6432NodeInterwovenWorksiteClientCommonInstallRootbitness with a value of "X86"

Because I didn’t have any iManage products installed at the time, it was safe for me to delete the entire HKLM\SOFTWARE\WOW6432Node\Interwoven reg key.

The installer then ran successfully after this. Thank goodness for Sysinternals by Mark Russinovich.

Reset the CSC (Offline Files) database

Over the years I’ve had many issues where a Windows client PC just won’t connect to a share using the FQDN, but can connect using the NetBios name. There have also been plenty of occasions where the opposite is true, too.

I had the issue again today on a freshly-built Windows 10 machine. Folder redirection wouldn’t apply because the user’s desktop folder was inaccessible.

Then I came across this post on the Spiceworks forums that mentioned resetting the CSC database. I tried that, and folder redirection now works perfectly!

If caching is enabled on the share in question, and the CSC database is knackered, you’ll run into this same problem.

If only I’d known about this earlier in my 15-year IT career, it could potentially have saved a lot of headaches.

Add the following reg key/value. then reboot the PC:

reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Csc\Parameters /v FormatDatabase /t REG_DWORD /d 1 /f

Retrieving voicemail files from a Mitel 3300 controller, and converting them to WAV

Today I had the displeasure of having to figure out how to retrieve and convert voicemail files from some old Mitel 3300 controllers.

Thanks to this forum post which pointed me in the right direction.

  1. Connect via FTP using a proper FTP client like FileZilla, to the 3300’s IP address
  2. Navigate to /vmail/d/vm/grp/<extension>
  3. Grab the relevant file. They’re all G.711 U-Law format RAW audio files
  4. Grab SoX – https://sourceforge.net/projects/sox/, and extract it somewhere
  5. Run the following command:
sox.exe --channels 1 --type raw --rate 8000 -e u-law -v 1 -D <inputfile> outputfile.wav

Quickly uninstall an MSI on multiple computers using WMI

Today I was working on reducing our vulnerability attack surface, and needed to remove Adobe Reader from our servers. It appears that it was installed as part of a VM image, but never maintained afterwards.

Long story short, rather than mess around with ConfigMgr baselines or Applications, I decided to go the direct route. To top it off, PowerShell remoting’s currently playing up. I ended up using WMI via the method that I outlined in my previous post.

Given an array of server names in $servers:

$servers | %{Invoke-WmiMethod -Class Win32_Process -Name Create -ArgumentList 'MsiExec.exe /x "{AC76BA86-7AD7-1033-7B44-AA1000000001}" /norestart /qn' -ComputerName $_}

Trigger a remote GPUpdate without PSRemoting or PSExec

I recently enabled Windows Firewall on an unused server via GPO, but forgot to include the inbound RDP exception. This, of course, kicked me off my RDP session.

Rather than wait ~90 minutes for my revised GPO to take effect, I found that I could trigger a GPUpdate remotely using WMI (WinRM wasn’t enabled, and I didn’t want to use PSExec)

The following command does the trick:

Invoke-WmiMethod -Class Win32_Process -Name Create -ArgumentList "gpupdate.exe" -ComputerName <computername>

Correct Horse Battery Staple for PowerShell, AKA Random Memorable Password Generator

I’m a fan of using correcthorsebatterystaple.net for generating temporary passwords for users, rather than always using a static password. The site itself is a reference to the XKCD webcomic, and yes, I’m aware that there are plenty of opinions on the web about this topic.

password_strength

I’ve had the idea in the back of my mind for a while to see if I could replicate the site’s functionality in PowerShell. I noticed that the source code for the site is on GitHub, so I ducked over there to check out the word list.

I found that it’s possible to replicate most of the functionality of the site with just two lines of PowerShell (although it doesn’t result in very readable code):

  1. I used Invoke-WebRequest to grab the word list from GitHub
  2. I then expanded out the Content property, and split it up given the comma delimiter
  3. I then used a combination of the Range operator, Foreach-Object, [string]::join,Get-Random and the TextInfo class to generate a given number of passwords along these rules:
    1. 4 random words, each with the first letter capitalised
    2. A separator in between
    3. A random number between 1 and 99 at the end

Note that this isn’t failure-proof, and isn’t intended to be used in any complex scenario. There’s no error handling, and not much flexibility built in. It’s just a quick function you could put into your PowerShell profile.

You can, at least, do the following:
A PowerShell window displaying the output of Get-RandomPassword. Also shows the function being called with a Count parameter as well as a separator parameter

Here’s the code:

function Get-RandomPassword {
[OutputType([string])]
Param
(
[int]
$Count = 1,
[string]
$Separator = ''
)
$words = (Invoke-WebRequest 'https://bitbucket.org/jvdl/correcthorsebatterystaple/raw/773dbccc9b9e1320f076c432d600f19785c41792/data/wordlist.txt' | Select-Object ExpandProperty Content).Split(',')
1..$Count | ForEach-Object {"$([string]::Join($Separator,(1..4 | ForEach-Object {[cultureinfo]::CurrentCulture.TextInfo.ToTitleCase(($words | Get-Random))})))$Separator$(1..99 | Get-Random)"}
}

ADFS 3.0 Error: The Web request failed because the web.config file is malformed

Had a strange one today after an Azure outage. One of my Server 2012 R2 ADFS proxies wouldn’t start the ADFS service.

When looking in the logs, it appeared like a case of simply having to re-establish the proxy trust, but I got a different error when trying to start the service:

The federation server proxy could not be started.
Reason: Error retrieving proxy configuration from the Federation Service.

Additional Data
Exception details:
An error occurred when attempting to load the proxy configuration.

There were other errors in the ADFS Event logs about a malformed config file:

The Web request failed because the web.config file is malformed.

User Action:
Fix the malformed data in the web.config file.

Exception details:
Root element is missing. (C:\Windows\ADFS\Config\microsoft.identityServer.proxyservice.exe.config)
Root element is missing.

When I opened the abovementioned config file, it was empty. I compared this to the config file on the other ADFS proxy, and that one looked like a normal config file.

My solution, and what ended up fixing the issue in the end, was simply to copy the contents of the .config file from the working ADFS proxy to the broken one. I could then re-establish the proxy trust, and everything started running again.

I’m not sure if this would work, but in case you don’t have another ADFS proxy to grab the config file from, here’s a sanitised version of mine:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="microsoft.identityServer.proxyservice" type="Microsoft.IdentityServer.Management.Proxy.Configuration.ProxyConfiguration, Microsoft.IdentityServer.Management.Proxy, Version=6.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL" />
</configSections>
<microsoft.identityServer.proxyservice>
<congestionControl latencyThresholdInMSec="8000" minCongestionWindowSize="64"
enabled="true" />
<connectionPool connectionPoolSize="200" scavengeInterval="5" />
<diagnostics eventLogLevel="15" />
<host tlsClientPort="49443" httpPort="80" httpsPort="443" name="adfs.example.com" />
<proxy address="" />
<trust thumbprint="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
proxyTrustRenewPeriod="21600" />
</microsoft.identityServer.proxyservice>
<!-- <system.serviceModel>
<diagnostics>
<messageLogging logEntireMessage="true"
logMessagesAtServiceLevel="true"
logMessagesAtTransportLevel="true">
</messageLogging>
</diagnostics>
</system.serviceModel> -->
</configuration>

Once I’d resolved the problem, I did a bit of searching around for this error message, and it appears that other people have had the same problem previously, with no resolution listed in the one thread that I looked at on the TechNet forums.

Script to create new local admin account for use with LAPS

I’ve got a bunch of older SOE machines that still had the local Administrator account enabled. As part of implementing Microsoft LAPS, I wanted to disable that account, and use a newly-created ‘LocalAdmin’ account with LAPS.

The account is created with a randomly-generated GUID as the password. The account’s password is going to come under the management of LAPS anyway. Additionally, it would be a terrible idea to hard-code a password into a script that’s stored in Sysvol.

If an account with that name already exists, the script will quit. Some basic events are also logged to the Event Log to indicate what happened.

My first revision of the script used ADSI to create the account and add it to the Administrators group, but I found that my mileage varied with that method. Some computers had the account created, but it wasn’t a member of Administrators.

It’s now set up to use plain “NET USER” and “NET LOCALGROUP” commands. This is an example of what would be executed:
2016-04-04 16_01_06

This script is designed to be set up as a computer Startup Script:

# The name of the account
$accountName = 'LocalAdmin'
$accountFullName = 'Local Administrator'
$accountComment = 'Backup Local Administrator Account'
# Any users listed here will be disabled by this script
$usersToDisable = 'Administrator','Guest'
# Set up some Event Log stuff
$sourceName = "$($MyInvocation.MyCommand.Name).ps1"
New-EventLog LogName Application Source "$sourceName" ErrorAction SilentlyContinue WarningAction SilentlyContinue
# If the account already exists, exit
if ((Get-WmiObject Win32_UserAccount filter "domain = '$Env:COMPUTERNAME' and Name = '$accountName'") -ne $null) {
Write-EventLog LogName Application Source $sourceName EntryType Information EventId 1 Message "$accountName already exists" ErrorAction SilentlyContinue WarningAction SilentlyContinue
exit
}
# Create the account
cmd.exe /c "net user $accountName `"$([guid]::NewGuid().guid)`" /add /y /comment:`"$accountComment`" /fullname:`"$accountFullName`""
# Add the account to the Administrators group
cmd.exe /c "net localgroup Administrators $accountName /add"
# Disable the specified users
$usersToDisable | Foreach-Object {cmd.exe /c "net user $_ /active:no"}
# Try and write an event to the Event Log
Write-EventLog LogName Application Source $sourceName EntryType Information EventId 2 Message "Created local administrator account: $accountName" ErrorAction SilentlyContinue WarningAction SilentlyContinue