PowerShell: Download a list of files

I was catching up on my Twitter feeds for the day, and came across this question by Troy Hunt, the Aussie IT security guru:

//platform.twitter.com/widgets.js

There were some useful replies, but also a few of the “I’m a *nix user, haha I scoff at your Windows user problem” responses. They quoted using wget and curl to read a text file full of URLs as their solutions.

//platform.twitter.com/widgets.js

That got me thinking, as wget and curl are used as aliases in PowerShell nowadays for the Invoke-WebRequest cmdlet.

Unfortunately it’s not as simple as using wget in *nix, as Invoke-WebRequest (or ‘iwr’ for short) does more than simply download files. It returns a Microsoft.PowerShell.Commands.HtmlWebResponseObject.

This makes Invoke-WebRequest incredibly powerful and useful for a good deal more than just downloading files. If this wasn’t the case, the syntax would be simpler than the *nix example tweeted above:

gc urls.txt | % {iwr}
(This will download the files, but not save them to disk)

There are plenty of examples around on the ‘net where people have developed full-blown functions, or used BITS, or the .NET System.Net.WebRequest class. I’ve done the same myself previously, but I wanted a one-liner that got as close to the traditional *nix command.

The thing with downloading files with Invoke-WebRequest is that you need to specify an –outfile parameter in order to save them to disk.

The closest I could get was the following one-liner. Definitely not something you’d type every day:

gc urls.txt | % {iwr $_ -outf $(split-path $_ -leaf)}

Or in long-form:

Get-Content urls.txt | ForEach-Object {Invoke-WebRequest $_ -OutFile $(Split-Path $_ -Leaf)}

Because Invoke-WebRequest doesn’t return the filename, we need to make use of the filename in the URL that we already have on file. This is done with the Split-Path cmdlet, specifically with the –Leaf switch to get the last part of it.

If anyone knows a better way to do it in native PowerShell, please leave a comment below. I’d love to know, and learn!

As for Troy Hunt’s original question, I use DownThemAll, a Firefox add-on to queue and download files. It has worked well for me for years. It had great retry options, and can handle authenticated URLs. Plus, it can be used to scrape pages for certain filetypes and “down” them all.

One thought on “PowerShell: Download a list of files

  1. This was a big help. I searched and searched for a way to do this and did come across DownloadThemAll, but from what I could gather it could only batch download if the requested urls came in a numerical sequence. file.com/file1, file.com/file 2, and so on. But I’m trying to grab pages from wiktionary based on a text file with the words I want to save the dictionary entry for. Wiktionary.com/Mouse, Wiktionary.com/Villa, and so on. I originally started with google sheet and importxml, but when you make too many of such requests something seems to go wrong in the spreadsheet. You method seems to work very well with my sample of 10 or so words… and I get html data I need saved locally. Hopefully then I can use importxml to grab the local files and not run into issues of an overloaded serve. Just wanted to say, appreciate your writing this article. Cheers. 🙂

    Like

Leave a Reply to Graham Dahl (@GrahamDahl1) Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s