PowerShell: Download a list of files

I was catching up on my Twitter feeds for the day, and came across this question by Troy Hunt, the Aussie IT security guru:

//platform.twitter.com/widgets.js

There were some useful replies, but also a few of the “I’m a *nix user, haha I scoff at your Windows user problem” responses. They quoted using wget and curl to read a text file full of URLs as their solutions.

//platform.twitter.com/widgets.js

That got me thinking, as wget and curl are used as aliases in PowerShell nowadays for the Invoke-WebRequest cmdlet.

Unfortunately it’s not as simple as using wget in *nix, as Invoke-WebRequest (or ‘iwr’ for short) does more than simply download files. It returns a Microsoft.PowerShell.Commands.HtmlWebResponseObject.

This makes Invoke-WebRequest incredibly powerful and useful for a good deal more than just downloading files. If this wasn’t the case, the syntax would be simpler than the *nix example tweeted above:

gc urls.txt | % {iwr}
(This will download the files, but not save them to disk)

There are plenty of examples around on the ‘net where people have developed full-blown functions, or used BITS, or the .NET System.Net.WebRequest class. I’ve done the same myself previously, but I wanted a one-liner that got as close to the traditional *nix command.

The thing with downloading files with Invoke-WebRequest is that you need to specify an –outfile parameter in order to save them to disk.

The closest I could get was the following one-liner. Definitely not something you’d type every day:

gc urls.txt | % {iwr $_ -outf $(split-path $_ -leaf)}

Or in long-form:

Get-Content urls.txt | ForEach-Object {Invoke-WebRequest $_ -OutFile $(Split-Path $_ -Leaf)}

Because Invoke-WebRequest doesn’t return the filename, we need to make use of the filename in the URL that we already have on file. This is done with the Split-Path cmdlet, specifically with the –Leaf switch to get the last part of it.

If anyone knows a better way to do it in native PowerShell, please leave a comment below. I’d love to know, and learn!

As for Troy Hunt’s original question, I use DownThemAll, a Firefox add-on to queue and download files. It has worked well for me for years. It had great retry options, and can handle authenticated URLs. Plus, it can be used to scrape pages for certain filetypes and “down” them all.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s