Powershell Scripts - Get-Parse & Publish-Parse - upload and download data from Parse.com Core DB

If you haven't checked out Parse as a hosted database platform for your website and/or app, you should.  They are free to start out and when your site or app takes off, they scale up with your traffic.  It's great for startups.  I didn't see a connector in Powershell to upload/download data from the core DB, so I made a pair of functions to do just that. They require Invoke-RestMethod which requires Powershell version 3 or later.

Lets try to upload a very simple table of 10 items, 2 columns:

PS C:\> $Data | FT -AutoSize

   code available
   ---- ---------
4993918      True
5449831      True
9565284      True
9251223      True
6062778      True
5107628      True
3599651      True
6477287      True
7101669      True
8115872      True


PS C:\> Publish-Parse -Class 'Codes' -AppID $AppID -APIKey $APIKey -Data $Data

createdAt                objectId  
---------                --------  
2015-06-29T23:42:14.104Z pmgrk5M87c
2015-06-29T23:42:14.330Z ivWv0Va6vE
2015-06-29T23:42:14.552Z Srj9Rcsvek
2015-06-29T23:42:14.766Z 4Muyx5v3kc
2015-06-29T23:42:14.978Z 6o4UrWc12J
2015-06-29T23:42:15.209Z 4EKTi7ZkTl
2015-06-29T23:42:15.423Z 6LMOBaSlbw
2015-06-29T23:42:15.646Z VzEgWXYbCV
2015-06-29T23:42:15.858Z YRIyxUD3Ue
2015-06-29T23:42:16.080Z UJgd2vfP5Y

I can see the new table in my web data browser in Parse:

Lets try to get that data back. Notice how Parse adds extra fields; if you don't want this in your data, use Select -ExcludeProperty to remove them.

PS C:\> Get-Parse -Class 'Codes' -AppID $AppID -APIKey $APIKey  | FT -AutoSize

available    code createdAt                objectId   updatedAt               
---------    ---- ---------                --------   ---------               
     True 4993918 2015-06-29T23:42:14.104Z pmgrk5M87c 2015-06-29T23:42:14.104Z
     True 5449831 2015-06-29T23:42:14.330Z ivWv0Va6vE 2015-06-29T23:42:14.330Z
     True 9565284 2015-06-29T23:42:14.552Z Srj9Rcsvek 2015-06-29T23:42:14.552Z
     True 9251223 2015-06-29T23:42:14.766Z 4Muyx5v3kc 2015-06-29T23:42:14.766Z
     True 6062778 2015-06-29T23:42:14.978Z 6o4UrWc12J 2015-06-29T23:42:14.978Z
     True 5107628 2015-06-29T23:42:15.209Z 4EKTi7ZkTl 2015-06-29T23:42:15.209Z
     True 3599651 2015-06-29T23:42:15.423Z 6LMOBaSlbw 2015-06-29T23:42:15.423Z
     True 6477287 2015-06-29T23:42:15.646Z VzEgWXYbCV 2015-06-29T23:42:15.646Z
     True 7101669 2015-06-29T23:42:15.858Z YRIyxUD3Ue 2015-06-29T23:42:15.858Z
     True 8115872 2015-06-29T23:42:16.080Z UJgd2vfP5Y 2015-06-29T23:42:16.080Z


PS C:\> Get-Parse -Class 'Codes' -AppID $AppID -APIKey $APIKey  -objectID "pmgrk5M87c","VzEgWXYbCV" | FT -AutoSize

available    code createdAt                objectId   updatedAt               
---------    ---- ---------                --------   ---------               
     True 4993918 2015-06-29T23:42:14.104Z pmgrk5M87c 2015-06-29T23:42:14.104Z
     True 6477287 2015-06-29T23:42:15.646Z VzEgWXYbCV 2015-06-29T23:42:15.646Z

Source Code:

Powershell Script - Compare-SourceFiles - source code comparison between environments (prod, bcp, qa, dev)

I just built this little script this weekend while troubleshooting a release. We needed a way to compare the code between different environments as well as compare folder permission security within environments on different nodes in the farm. This function does exactly that, producing a nice report (use -ReportFile to output to a file as well) used to prevent change drift and environment variance. If the files are text files, a line by line comparison is also generated should the file hashes be different, which shows specific lines that are different. Binary files are still flagged as different should their MD5 hashes be different. When checking ACLs, effective file permissions are displayed for each source which can speed up troubleshooting permissions issues.

PS c:\> Compare-SourceFiles -Source1 \\server1\c$\prod -Source2 \\server2\c$\dev -ACLs

WARNING: Files missing from '\\server1\c$\prod': 3
\FrameXML.log
\gx.log
\Sound.log



WARNING: File Hash different: '\a.exe'


WARNING: ACL Access Permissions different: '\a.exe'
   File Permissions: '\\server1\c$\prod\a.exe'

FileSystemRights AccessControlType IdentityReference      IsInherited InheritanceFlags PropagationFlags
---------------- ----------------- -----------------      ----------- ---------------- ----------------
     FullControl             Allow BUILTIN\Administrators       False             None             None
     FullControl             Allow NT AUTHORITY\SYSTEM          False             None             None
     FullControl             Allow DOMAIN\User1                 False             None             None



   File Permissions: '\\server2\c$\dev\a.exe'

           FileSystemRights AccessControlType IdentityReference                IsInherited InheritanceFlags PropagationFlags
           ---------------- ----------------- -----------------                ----------- ---------------- ----------------
                FullControl             Allow BUILTIN\Administrators                 False             None             None
ReadAndExecute, Synchronize             Allow Everyone                               False             None             None
        Modify, Synchronize             Allow NT AUTHORITY\Authenticated Users       False             None             None
                FullControl             Allow NT AUTHORITY\SYSTEM                    False             None             None





WARNING: File Hash different: '\connection.log'

InputObject                                                                                    SideIndicator
-----------                                                                                    -------------
2/16 16:02:39.744  GRUNT: state: LOGIN_STATE_CONNECTING result: LOGIN_OK                       =>                  
2/16 16:02:39.946  GRUNT: state: LOGIN_STATE_AUTHENTICATING result: LOGIN_OK                   =>           
2/16 16:02:40.128  GRUNT: state: LOGIN_STATE_CHECKINGVERSIONS result: LOGIN_OK                 =>           
2/16 16:02:40.205  GRUNT: state: LOGIN_STATE_HANDSHAKING result: LOGIN_OK                      =>           
2/16 16:02:40.375  GRUNT: state: LOGIN_STATE_AUTHENTICATED result: LOGIN_OK                    =>           
2/16 16:02:40.375  ClientConnection Initiating: COP_CONNECT code=CSTATUS_CONNECTING            =>           
2/16 16:02:40.708  ClientConnection Completed: COP_CONNECT code=RESPONSE_CONNECTED result=TRUE =>           
2/16 16:02:40.724  ClientConnection Initiating: COP_AUTHENTICATE code=CSTATUS_AUTHENTICATING   =>           
2/16 16:02:41.311  ClientConnection Completed: COP_AUTHENTICATE code=AUTH_OK result=TRUE       =>           
2/16 16:02:41.780  ClientConnection Initiating: COP_GET_CHARACTERS code=43                     =>           
2/16 16:02:42.286  ClientConnection Completed: COP_GET_CHARACTERS code=44 result=TRUE          =>           
2/16 16:02:59.661  GRUNT: state: LOGIN_STATE_DISCONNECTED result: LOGIN_OK                     =>           
2/16 16:03:03.816  ClientConnection Initiating: COP_LOGIN_CHARACTER code=77                    =>           
2/16 16:03:04.237  ClientConnection Completed: COP_LOGIN_CHARACTER code=78 result=TRUE         =>           
2/16 16:01:53.409  GRUNT: state: LOGIN_STATE_CONNECTING result: LOGIN_OK                       <=           
2/16 16:02:14.449  GRUNT: state: LOGIN_STATE_FAILED result: LOGIN_CONVERSION_REQUIRED          <=           
2/16 16:02:14.453  GRUNT: state: LOGIN_STATE_FAILED result: LOGIN_CONVERSION_REQUIRED          <=           





WARNING: File Hash different: '\cpu.log'

InputObject                                                                         SideIndicator
-----------                                                                         -------------
2/16 16:02:30.896  vendor: 1                                                        =>           
2/16 16:02:30.896  features: 00000397                                               =>           
2/16 16:02:30.896  cores: 2                                                         =>           
2/16 16:02:30.896  threads: 4                                                       =>           
2/16 16:02:30.896  vendor id string= GenuineIntel                                   =>           
2/16 16:02:30.896  standard (13): 1b=02100800 1c=7FDAFBBF 1d=bfebfbff 4a=1C004121   =>           
2/16 16:02:30.896  extended (8): 1c=00000021 1d=2c100000 8c=00000000                =>           
2/16 16:02:30.896  processor brand string= Intel(R) Core(TM) i5-4670K CPU @ 3.40GHz =>           
2/16 16:01:42.716  vendor: 1                                                        <=           
2/16 16:01:42.716  features: 00000397                                               <=           
2/16 16:01:42.716  cores: 2                                                         <=           
2/16 16:01:42.716  threads: 4                                                       <=           
2/16 16:01:42.716  vendor id string= GenuineIntel                                   <=           
2/16 16:01:42.716  standard (13): 1b=00100800 1c=7FDAFBBF 1d=bfebfbff 4a=1C004121   <=           
2/16 16:01:42.716  extended (8): 1c=00000021 1d=2c100000 8c=00000000                <=           
2/16 16:01:42.716  processor brand string= Intel(R) Core(TM) i5-4670K CPU @ 3.40GHz <=           






==============================
File Source Comparison Summary
==============================

Path              FileCount MissingFiles SharedFiles DifferentHashes DifferentACLs
----              --------- ------------ ----------- --------------- -------------
\\server1\c$\prod         3            3           3               3             1
\\server2\c$\dev          6            0           3               3             1

How-To - Name your functions correctly

When you name functions, there is a set standard which you must follow or you'll see this error if you try and put your function into a module and load it:

WARNING: Some imported command names include unapproved verbs which might make them less discoverable. Use the Verbose parameter for more detail or type Get-Verb to see the list of approved verbs.

That format is the basic combination of a Verb and a Noun joined by a dash ("-").  The Noun can be an alphanumeric string, no spaces and a limited scope of special characters (underscore is ok).  The Verb however, has to be on a specific list.  Lets take a look at the valid list:

PS C:\> Get-Verb | Format-Wide -AutoSize


Add          Clear        Close        Copy         Enter        Exit        Find        Format     
Get          Hide         Join         Lock         Move         New         Open        Optimize   
Pop          Push         Redo         Remove       Rename       Reset       Resize      Search     
Select       Set          Show         Skip         Split        Step        Switch      Undo       
Unlock       Watch        Backup       Checkpoint   Compare      Compress    Convert     ConvertFrom
ConvertTo    Dismount     Edit         Expand       Export       Group       Import      Initialize 
Limit        Merge        Mount        Out          Publish      Restore     Save        Sync       
Unpublish    Update       Approve      Assert       Complete     Confirm     Deny        Disable    
Enable       Install      Invoke       Register     Request      Restart     Resume      Start      
Stop         Submit       Suspend      Uninstall    Unregister   Wait        Debug       Measure    
Ping         Repair       Resolve      Test         Trace        Connect     Disconnect  Read       
Receive      Send         Write        Block        Grant        Protect     Revoke      Unblock    
Unprotect    Use

If you run Get-Verb by itself, you'll get a list that runs off the page, but also shows which group the verbs belong to.  For the most part, try to use the verb which mostly relates to the action that you are trying to do.  You can see what other functions do by simply looking at same verb function's help:

PS C:\> Get-Help wait* | Format-Table Name,Synopsis -AutoSize

Name          Synopsis                                                                              
----          --------                                                                              
Wait-Job      Suppresses the command prompt until one or all of the Windows PowerShell background...
Wait-Debugger Stops a script in the debugger before running the next statement in the script.       
Wait-Event    Waits until a particular event is raised before continuing to run.                    
Wait-Process  Waits for the processes to be stopped before accepting more input.                    

To see verbs from all commands that are currently loaded, try this little block.  I've marked it up the code to help explain what each line does.  Try running the first line (leaving off the pipe "|"), then the first and second lines, then the first 3 lines, etc.  You can see how passing the output of one cmdlet into the next works this way.

Get-Command |                      #Gets all commands from all loaded modules 
select -ExpandProperty Name |      #Expands the Name column
Where-Object{$_.contains("-")} |   #Only selects names containing the dash character 
ForEach-Object{$_.split("-")[0]} | #For each of the names, split it up into 2 objects on the dash, select only the 1st
select -Unique |                   #Remove duplicates
sort |                             #sort alphabetically
ConvertFrom-Csv -Header "Name" |   #Converts a string list into an object list
Format-Wide -AutoSize              #Displays the list in a multi-column list

Lastly, if you are planning on releasing a module with a bunch of functions, try naming them in a similar way to indicate they are all from your module.  Most third party modules do this.  The format should be Verb-PrefixNoun.  For example, the NetApp DataONTAP Powershell module uses the prefix "Na", so there commands look like this:

• Get-NaLun
• Set-NaVol
• Restore-NaHostFile

Quick Script - Test-PingRDP - Check if servers are pingable and have RDP enabled

I answered a question on a forum with a similar version of this script.  I made it a function instead of a lose script, which have some negatives comparatively.  Functions also are much easier to wrap up into a module for easy distribution.  PSGet/OneGet also requires everything in modules.

Example:

PS C:\> @"
Server1
Server2
Server3
Server4
NotAServer
"@ | Set-Content servers.txt

PS C:\>Test-IPRDP -ServerList servers.txt

PS C:\>Import-Csv .\ServerStatus.csv

ServerName TimeStamp           RDPPort Results
---------- ---------           ------- -------
server1    2015-06-26T09:35:51 3389    Up     
server2    2015-06-26T09:35:52 3389    Up     
server3    2015-06-26T09:35:53 3389    Up     
server4    2015-06-26T09:35:54 3389    Up     
NotAServer 2015-06-26T09:35:55 3389    Down     

PS C:\>

Code:

Beginner's Guide - Powershell for the newbie

Whether you're new to the world of Windows administration or you've been living under a rock for the past 8 years, it's OK not to know about perhaps the single most effective tool for administrating Windows environments since Active Directory.  I'm not going to reinvent the wheel in this blog entry as there are tons of great resources out there, but I am going to provide some of my favorites and recommendations.

Above all, in order to learn how to use Powershell, you must practice.  Most commands are innocuous and safe to run on your workstation or laptop; but to be safe, you are better off having a dedicated PC or virtual machine to create a sandbox environment.  I'll be creating a blog called "How-to: Building a Virtual Lab" soon and I'll add the link when it's up.

For now, lets get to the good stuff.

Getting Started

• Technet Windows Powershell: Self Study Guide - I tell everyone to go here first.  This probably the best collection of intro links I know of.
• Microsoft Virtual Academy: Powershell Jump Start - If you like videos, this is the place to start.  Jason Helmick and Jeffery Snover (inventor of Powershell) get you right into the shell and using it in minutes.
• Don Jones Powershell Crash Course - Don't have time to watch the MVA video above? Check this out.
• Powershell Pro Tutorials - A good blog that I first used to get started (it's a bit dated)
• Windows IT Pro - A really great blog for enabling Powershell to run scripts. By default, running unsigned scripts is disabled, this shows you have to get around that.

Building the knowledge

• Powershell Tutorial Online - Another excellent article series which has valuable under the hood fundamentals for making your scripts really do amazing things.
• MVA: Advanced Tools and Scripting - The second part which is a must see.

Coming to Windows?

• Powershell for Unix people - Many unix commands work in a similar way to bash, though not case sensitive thank goodness.
• Powershell on Mac - well editing on a mac anyways.  Powershell over SSH will come soon enough and then you'll be able to run commands on a remote server from your non-windows client.

Books

• Learn Windows Powershell in a Month of Lunches - Don Jones & Jeffrey Hicks classic - most people learn powershell with this book.
• Learn Powershell Toolmaking in a Month of Lunches - Another Jones & Hicks book which I owe a lot of my more advanced knowledge to.
• Windows Powershell Cookbook - Lee Holmes offers a different perspective (through a developer's eyes) and has some creative tips for the IT admin.
• Powershell TFM (The Freak'n Manual) - My first Powershell book.

Classroom Learning

• Interface Powershell Training -  I took the Don Jones Master Class (boot camp) last year and he blew my mind in 5 days, despite my 6 previous years of Powershell experience. Worth the $3500 and they offer web based learning.  If you can get work reimbursement, all the better.
• Powershell.org Events - Many free webinars and if you are lucky, you might have a Powershell meetup group near you.

That's all for now. Did I miss something that you think should be on this list? Let me know in a comment below or Tweet me.

Powershell Script - Get-LastBootTime - remotely find out when your servers where last rebooted

Here's another handy and simple function which lets you query a bunch of servers at once to find out when they were last rebooted.  It's taking advantage of Get-WmiObject and the root\civ2\Win32_OperatingSystem class.  For maximum parallelization, I've forced pipeline usage to first colapse the process block into a single array to create only one Get-WmiObject call.  This is significantly faster than the default pipeline behavior which is sequential, with 1000 servers completing under a minute (local LAN) vs sequential calls taking close to 10 minutes.

Here's it in action:

PS C:\> Get-Content .\serverlist.txt | Get-LastBootTime

Name      UpSince
----      -------
NOTSERVER Unknown             
SERVER001 6/20/2015 10:46:25 PM
SERVER002 6/20/2015 10:26:24 PM
SERVER003 6/20/2015 10:30:17 PM
SERVER004 6/20/2015 10:27:52 PM
SERVER005 6/20/2015 10:38:27 PM
SERVER006 6/20/2015 10:30:12 PM
SERVER007 6/20/2015 11:54:01 PM
SERVER008 6/20/2015 10:28:39 PM
SERVER009 6/20/2015 10:26:15 PM
SERVER010 6/20/2015 10:27:01 PM

Source Code:

Powershell Script - ConvertTo-HtmlTable - making pretty html formatted tables for reports and emails.

Here's a nifty utility function I built long ago that really makes emailing reports or rendering web reports easy as pie (cake is a lie). You can even change the font color, background color, border color, border width, and cell padding around the text.

Here's a little example which displays the current directory contents and opens up your browser for you after the report's finished.

dir . | select Name,Length,Mode,LastWritetime | ConvertTo-HtmlTable | Set-Content .\test.html
Invoke-Item .\test.html

And the output in your browser:

Powershell Script - Invoke-Multithread - Make any script multithreaded

Here's a very useful wrapper which helps you turn your single threaded sequential functions into multithreaded beasts which complete in a fraction of the time when addressing many servers (or whatever network devices).

The function works by taking your code as a [scriptblock] object and a list of computers (servers, IPs, whatever) and breaking up the list based on a throttle limit (default is 25) and starting powershell jobs.  The script waits for each job to finish and captures the return from the job and parses it into an object array and associates each computer's output with the name.

This function is great for generating reports, pushing configurations, making changes, etc, where native multithreaded cmdlets won't work and sequential code must be used.  Most admins new to powershell don't know how to leverage built in commands that automatically thread and this wrapper function does a nice job at speeding up slow scripts.

Here's an example where I'm just checking to see if the server is a windows server, but this could be any function which takes a string array "ComputerName" as a parameter.

And the output:

ComputerName  Output
------------  ------
SERVER1       Is a Windows Server
SERVER2       Is a Windows Server
SERVER3       Is a Windows Server
SERVER4       Is a Windows Server
SERVER5       Is a Windows Server
SERVER6       Is a Windows Server
SERVER7       Is a Windows Server
SERVER8       Is a Windows Server
SERVER9       Is a Windows Server
SERVER10      Is a Windows Server
SERVER11      Is a Windows Server
SERVER12      Is a Windows Server
SERVER13      Is a Windows Server
SERVER14      Is a Windows Server
SERVER15      Is a Windows Server
SERVER16      Is a Windows Server
SERVER17      Is a Windows Server
SERVER18      Is a Windows Server
SERVER19      Is a Windows Server
SERVER20      Is a Windows Server
SERVER21      Is a Windows Server
SERVER22      Is a Windows Server
SERVER23      Is a Windows Server
SERVER24      Is a Windows Server
SERVER25      Is a Windows Server
SERVER26      Is a Windows Server
SERVER27      Is a Windows Server
SERVER28      Is a Windows Server
SERVER29      Is a Windows Server
SERVER30      Is a Windows Server
SERVER31      Is a Windows Server
SERVER32      Is a Windows Server
SERVER33      Is a Windows Server
SERVER34      Is a Windows Server
SERVER35      Is a Windows Server
SERVER36      Is a Windows Server
SERVER37      Is a Windows Server
SERVER38      Is a Windows Server
SERVER39      Is a Windows Server
SERVER40      Is a Windows Server
SERVER41      Is a Windows Server
SERVER42      Is a Windows Server
SERVER43      Is a Windows Server
SERVER44      Is a Windows Server
SERVER45      Is a Windows Server
SERVER46      Is a Windows Server
SERVER47      Is a Windows Server
SERVER48      Is a Windows Server
SERVER49      Is a Windows Server
SERVER50      Is a Windows Server
NAS1          Is NOT a Windows Server

Powershell script: Cleaning up C:\Windows\Installer directory - the correct way

This very simple script is built off Heath Stewart's VB Script which identifies the files linked for installed products that need to be kept.  My powershell wrapper simply takes his output and automates the deleting of what's not needed.  Usage is simple: drop the script in a temp directory, and run. It will create and run Heath's script which creates another file (output.txt) which is then read back in and parsed.  You'll see what's being removed and what's being kept.

It can be pushed as a scriptblock to remote servers using PS Remoting or PSexec.exe.

Enjoy!

Hello World!

Hello there fellow techies, scripters, IT professionals, developers.  I'm Bryan, Systems Architect and automation guru specializing in Powershell with experience with most of Microsoft technologies (see my Linkedin profile for more details).

The purpose of this blog is to pay back the 15+ years of professional experience I've acquired as much of my knowledge was built on blogs like these.  With Microsoft's shift towards an open source model, I thought it'd be a great time to open up my library of over 2000 scripts.  I'll be documenting how-to's as I build out new environments and labs as well as sharing some of my scripts and modules.

The usual disclaimer: I'm not responsible for anything you break by running my scripts or following my how-to's.  Please use caution and take time to understand what you are doing before you run something (I'll link technet as much as possible).  Always test in a non-production environment if possible.  My code is free to use, but please give credit.  Materials on this blog or any of my code repositories may not be republished without explicit written authorization from me.