Sunday, May 12, 2019

Powershell Script - MassDownloader - Efficient, Automated, Fault Tolerant, idempotent downloader with real time metrics

So it's been exactly 1 year since I published here. Life has been beyond busy. I'm hoping to get back on the bandwagon of updating my blog, I have dozens of functions and scripts I'd like to publish that I've been building for various needs.  I'll also hopefully get some how-to's posted about running windows server 2019 VM host with NAS & other VMs.

Recently, I needed to download a ton of files and I found trying to do it with a browser or wget wasn't going to work.  I wanted a robust, lightweight, efficient tool which I could easily add custom logic URL parsing logic to.  I wanted it to be idempotent while supporting resumable downloads that run in the background.  I decided to leverage the BITS platform which is really the best of breed downloading method on windows.

I also wanted to see real time statistics as it churned through hundreds of downloads.   I also wanted the function to run independent of the scrapper function which was populating the download list file with all the URLs to download.  This way I had a means of throttling downloads that efficiently took advantage of my full bandwidth without over saturating it with excess overhead and packet loss.

The solution I came up with leaves out some more advanced features like auto throttling based on tcp statistics and I took a stab at calculating estimated total time of completion and I found without better accounting, the results were garbage.  I have a few other ideas I might implement in a future version of this, but I figured I'd just publish what I used to download hundreds of files, over 200gb total successfully.

The script also includes a way to stop all downloads should you need to.  Just run "Stop-DownloadFiles"

Ran from ISE:


Ran from standard prompt:



Here's the source code:

Saturday, May 12, 2018

Powershell Script - Invoke-RemoteShellCommand - Run remote Linux bash commands with output wrapper

Recently, I've been working much more with linux servers and I even challenged myself to run Ubuntu on my primary personal laptop while still doing mostly powershell development.  I needed a way to quickly scale powershell core deployment out to servers, so I came up with a little wrapper function which simplifies this task leveraging putty's plink ssh client.  Her'e's what I came up with.



Powershell Script - 7zip/unzip powershell native replacement

Last year, I was tasked with creating a replacement for unzip executables in our environment with native powershell/.net extraction function. Because our environment used to heavily rely on both 7z.exe and unzip.exe (code has been updated so it's not anymore), I made a series of functions which mimic a limited set of parameter behavior for these executables. Here's what I came up with.



Thursday, December 14, 2017

State of Net Neutrality - End of Internet Freedom?

Today is a very dark day for internet freedom.  With the Federal Communications Commission voting today to repeal the Obama era protection from two years ago, an assault has begun on our freedom of speech.  It won't be long before we'll start to see ISPs and telecoms throttle various sites and services.

As time goes on, new campaigns to get customers to switch over to tiered plans will begin and seem enticing at first, offering internet for cheaper prices.  But there will be catches, and they will get worse with time.  More restrictions and higher priced tiers will become normal, just offering the same internet we currently have now.

Eventually, everyone will be forced to switch over to these new tiered plans, ultimately with ISPs and telecoms making billions more off the public and businesses. This is especially true for the providers who have market monopolies, like the big four cellular networks and the cable companies in the US.

At the same time, censorship will become more obvious, as emboldened and greedy providers become like mobsters, collecting "safety" money from services, websites, and content providers just so their site or service isn't censored or throttled.   Free VoIP or chat providers will have to pay ransom money to the providers or no one will be able to get to their services, so they will disappear.

The darkweb as we know it will likely be censored and blocked entirely.  A cat and mouse game will begin with hackers and activists trying to circumvent an ever growing firewall controlled by the providers (and likely the government as well) meant maximize profit while allowing propaganda to find it's way into everything we do online.

While this future is very dark indeed, there will be resistance.  First, write and call your congressman, they can overturn this vote.  The Electronic Frontier Foundation will be the spear tip, with it's staff of digital expert attorneys suing the FCC.  Also, I expect decentralized wireless mesh networks groups to see a surge in growth, with many new nodes coming online as the resistance and knowledge spreads.

While EFF and other groups fight the FCC, most people won't have the skills or will be too geographically far to participate in a wireless mesh networking group, there is something that almost everyone can do to combat throttling and censorship by their ISP and wireless provider.  They can use a VPN provider to tunnel their data so the providers can't see it.  This also prevents them from monitoring your usage (which is something they've been able to do, even before the vote today).

I signed up for PureVPN, 3 years for $69, which is a really good deal. You get to use 5 devices simultaneously, but if you configure your home router, all devices connected to it will be secured and it only counts as one device.  The Android and iOS apps works great from my testing, but the best speed is had by the in-browser plug-ins for both chrome and firefox.  They have plenty of servers in the US and abroad in other countries too.

So I encourage everyone who believes in net neutrality to donate to EFF, pick a VPN provider and start using it to connect to the internet, and if you are more savvy and have the know-how, join a local wireless mesh networking group and set up your own node. If you have the resources, getting your HAM license will let you operate a HAMnet, which will allow you to legally use radio frequencies that can carry wireless connections hundreds of miles.

I predict it will have to get worse, before it gets better.  Meanwhile, the stocks of ISPs & wireless providers will likely grow significantly as they are the real winners today.

Wednesday, May 17, 2017

Powershell Video - Jeffrey Snover - State of the Union

Very much worth watching, but TLDR; learning Powershell will make you successful, even if you are a linux engineer.