Powershell Script - MassDownloader - Efficient, Automated, Fault Tolerant, idempotent downloader with real time metrics

So it's been exactly 1 year since I published here. Life has been beyond busy. I'm hoping to get back on the bandwagon of updating my blog, I have dozens of functions and scripts I'd like to publish that I've been building for various needs.  I'll also hopefully get some how-to's posted about running windows server 2019 VM host with NAS & other VMs.

Recently, I needed to download a ton of files and I found trying to do it with a browser or wget wasn't going to work.  I wanted a robust, lightweight, efficient tool which I could easily add custom logic URL parsing logic to.  I wanted it to be idempotent while supporting resumable downloads that run in the background.  I decided to leverage the BITS platform which is really the best of breed downloading method on windows.

I also wanted to see real time statistics as it churned through hundreds of downloads.   I also wanted the function to run independent of the scrapper function which was populating the download list file with all the URLs to download.  This way I had a means of throttling downloads that efficiently took advantage of my full bandwidth without over saturating it with excess overhead and packet loss.

The solution I came up with leaves out some more advanced features like auto throttling based on tcp statistics and I took a stab at calculating estimated total time of completion and I found without better accounting, the results were garbage.  I have a few other ideas I might implement in a future version of this, but I figured I'd just publish what I used to download hundreds of files, over 200gb total successfully.

The script also includes a way to stop all downloads should you need to.  Just run "Stop-DownloadFiles"

Ran from ISE:

Ran from standard prompt:

Here's the source code:

Powershell Quick Script - Wrap any powershell script into a batch file

It's been a number of months since I've written anything here which is unfortunate since I've been writing lots of nifty things in powershell for work. So this should be the first of many scripts which need sharing.

I recently ran into an issue where I had to make a single file script which could easily be ran by a simple user who could just double-click on it and it would just work, regardless of which version of powershell they had or if their Execution Policy was set correctly. The solution I went for was the embed a powershell script inside of a batch file which is more universally accepted on legacy systems and by most windows admins.

There are several solutions I found which involve encoding the powershell script into a long base64 string and feeding it into powershell.exe, but this has a size limitation which larger scripts easily hit. Another solution I saw was to strip away special formatting, comments, certain characters, and then wrap it in curly brackets "{}" and again feed it to powershell.exe as a command. This too also suffers from the max length problem as well as requires special editing to make it work.

My solution is much simpler, suffers from no length constraints and really has one drawback which matters if you are watching the error output stream. The solution is very simple. Just add the following line before your powershell script:

goto ExecutePowershell
cls

Then add the following after your script:

exit
#end of powershell code - batch code now:
:ExecutePowershell
echo off
set filename=%temp%\Tempscript.ps1
copy /y %0 %filename%
echo NOTE - you will see error output in the error stream about 'goto' - this is expected and can be ignored.
cls
powershell -ExecutionPolicy unrestricted -file %filename% %*
set /a el=%errorlevel%
del %filename%
exit %el%

Finally, save your script as a .cmd or .bat file extension.  What this will do is cause your code to be executed as batch, which will then copy itself to the temp location and add the extension ps1 and then feed that into powershell.exe while bypassing the execution policy ofthe machine. Once the powershell code finishes, it will exit, returning back to the batch wrapper which will then clean up the temp ps1 file and return the error code from powershell exit. I highly suggest your powershell code has it's own built in exits. Most automation systems are specifically looking for exit codes and you want exit with 0 if it's successful or another number if it's not. If you do don't want the script to close the window when it's done, replace the last line "exit %el%" with "pause". Here's an example of the complete file:

Powershell Script - Set-UserPassword - Remotely sets local account passwords

Here's a great script to change passwords in bulk on many servers. I've added verbose and error output for logging purposes as well as time/date stamping for when actual password setting occurs.

PS C:\> 'server1','server2','Badserver' | Set-UserPassword -Username 'test' -Password 'pass123' -Verbose
VERBOSE: Processing server 'server1'...
VERBOSE: Connected to server 'server1'...
VERBOSE: Retrieved user objects from server 'server1'...
VERBOSE: Found user 'test' from server 'server1'...
10/12/2015 14:41:31 - Successfully changed password for user 'test' on server 'server1'
VERBOSE: Processing server 'server2'...
VERBOSE: Connected to server 'server2'...
VERBOSE: Retrieved user objects from server 'server2'...
WARNING: ERROR: No user 'test' on server 'server2'
VERBOSE: Processing server 'Badserver'...
WARNING: ERROR: Failed to connect to server 'Badserver'
PS C:\>

Powershell Script - Push-Patch - remote installation for KB hotfix, MSI, MSU, or EXE files

Here's a wickedly useful function I threw together based on a strong need to push emergency patches to thousands of servers without the standard route of using a patching deployment system like SCCM, WSUS, Altiris, or HPSA.  All of these tools have their strengths and are generally considered superior to just manually pushing patches, but the sheer versatility of ease of being able to quickly deploy patches is what I required to make tight patching deadlines.

I recommend using this function with Get-HotFix, or Get-WmiObject -class Win32_product for a list of installed patches/software to determine if installation is needed and/or was successful.  You can also come in after the push and look at the log files on the servers in an automated fashion to see how things went.

This script does require PSRemoting to be enabled on target servers, which is true by default on all Windows Server 2012 R2 installs, but needs to be turned on either by GPO or by hand on previous versions.  There's a built in workaround if you pull down PSExec which will work on older servers without PSRemoting.

Powershell Script - Get-LocalUsers - Remotely query all local users and details

Here's a lovely script that runs with parallel pipeline queries using the [ADSI] .NET class to remotely query SAM to build nice object outputs which are great for security audits.  Here's some sample output:

PS C:\> 'Server1','Server2' | Get-LocalUsers


Server          : Server1
UserName        : Administrator
Active          : True
PasswordExpired : False
PasswordAgeDays : 16
LastLogin       : 6/3/2015 6:34:27 PM
Groups          : Administrators
Description     : Built-in account for administering the computer/domain

Server          : Server1
UserName        : Guest
Active          : False
PasswordExpired : False
PasswordAgeDays : 0
LastLogin       : 
Groups          : Guests
Description     : Built-in account for guest access to the computer/domain

Server          : Server2
UserName        : Administrator
Active          : True
PasswordExpired : False
PasswordAgeDays : 1
LastLogin       : 3/5/2015 7:28:14 PM
Groups          : Administrators
Description     : Built-in account for administering the computer/domain

Server          : Server2
UserName        : Guest
Active          : False
PasswordExpired : False
PasswordAgeDays : 0
LastLogin       : 
Groups          : Guests
Description     : Built-in account for guest access to the computer/domain


PS C:\> 

And the code:

Powershell Script - Get-ServerHDDinfo - Remotely pull hdd size info quickly

I wrote two versions of this function for different scenarios.  The first version is much faster but requires PS remoting enabled on all target servers.  the second version will work with any server with remote WMI enabled which means it will support older 2003 servers much more easily.

The core of the functions is the same: pull the Win32_Volume WMI class and build a report from hdd information.  The first function remotely executes a script block which builds it's own report and then sends it back to the calling computer, while the second version calls each computer's WMI directly and builds one report for all servers.  Even with Get-WmiObject's ability to query multiple servers at once, it still can't match the speed of Invoke-Command, which leverages the faster CIM framework.


Version 1 - Much faster Invoke-Command Version


Version 2 - Slower WMI

Powershell Script - Get-LastBootTime - remotely find out when your servers where last rebooted

Here's another handy and simple function which lets you query a bunch of servers at once to find out when they were last rebooted.  It's taking advantage of Get-WmiObject and the root\civ2\Win32_OperatingSystem class.  For maximum parallelization, I've forced pipeline usage to first colapse the process block into a single array to create only one Get-WmiObject call.  This is significantly faster than the default pipeline behavior which is sequential, with 1000 servers completing under a minute (local LAN) vs sequential calls taking close to 10 minutes.

Here's it in action:

PS C:\> Get-Content .\serverlist.txt | Get-LastBootTime

Name      UpSince
----      -------
NOTSERVER Unknown             
SERVER001 6/20/2015 10:46:25 PM
SERVER002 6/20/2015 10:26:24 PM
SERVER003 6/20/2015 10:30:17 PM
SERVER004 6/20/2015 10:27:52 PM
SERVER005 6/20/2015 10:38:27 PM
SERVER006 6/20/2015 10:30:12 PM
SERVER007 6/20/2015 11:54:01 PM
SERVER008 6/20/2015 10:28:39 PM
SERVER009 6/20/2015 10:26:15 PM
SERVER010 6/20/2015 10:27:01 PM

Source Code:

Powershell script: Cleaning up C:\Windows\Installer directory - the correct way

This very simple script is built off Heath Stewart's VB Script which identifies the files linked for installed products that need to be kept.  My powershell wrapper simply takes his output and automates the deleting of what's not needed.  Usage is simple: drop the script in a temp directory, and run. It will create and run Heath's script which creates another file (output.txt) which is then read back in and parsed.  You'll see what's being removed and what's being kept.

It can be pushed as a scriptblock to remote servers using PS Remoting or PSexec.exe.

Enjoy!