Asynchronous Processing using Powershell Jobs

John Wood — Wed, 17 Mar 2010 14:48:57 +0000

Powershell is a fantastic tool to use for management of multiple computers. I have been slowly converting many of our administrative functions from a hodge-podge mixture of CMD, BAT, VPS, Python, and Perl scripts. One daily administrative talks is copying a variety of backups from a variety of Windows servers to our data ’warehouse’ where they are then copied to tape.

I recently found the “Jobs” cmdlets in Powershell V2.0. With “Jobs” you can asynchronously process multiple tasks (e.g. copying backups from many remote machines to data storage on the current local machine). In my case, running the copies synchronously results in the process spanning over to the next morning. Backups are stored daily. If a backup is taken on Wednesday, I can’t have it being stored in our warehouse under Thursday.

Using Powershell “Jobs”, I can accommodate copying all the backups within the daily timeframe. However, I did wrestle with using “Jobs” at first. Mainly because I couldn’t see what happened with a JOB on my console! The trick is to make sure you issue a Receive-Job cmdlet after a JOB has finished.

Here a a couple of examples to try:

These example assume you are running Powershell V2.0 and you have enabled remoting on the machine your running this. Remoting does not have to be enabled on the remote servers but must be enabled on your local machine.

Begin with a simple copy you know works.

# Copies a directory and all files and sub-directories, I would
# suggest using something small to quickly see results
$source = "\\RemoteServer\d$\Backups # your remote path
$target = "W:\WoodyzTest"            # your local path
Copy-Item -Path $source -Destination $target -Recurse Verbose

Next, we’ll take the Copy-Item and put it in a Start-Job

$source = "\\RemoteServer\d$\Backups # your remote path
$target = "W:\WoodyzTest"            # your local path
$job Start-Job -Scriptblock {
Copy-Item -Path $source -Destination $target -Recurse -Verbose
}
Wait-Job $job  # For this test, we’ll wait for the job to complete

Run the above script, look at what is returned in the console, and then check to see of the copy worked.

Surprise! The directory was not copied! What happened? I have no idea because I did not see anything in the console. There were no error messages!

OK, let’s now add a Receive-Job

$source = "\\RemoteServer\d$\Backups # your remote path
$target = "W:\WoodyzTest"            # your local path
$job Start-Job -Scriptblock {Copy-Item -Path $source -Destination $target -Recurse -Verbose }
Wait-Job $job  # For this test, we’ll wait for the job to complete
Receive-Job $job  # will now display any results from the job

Now what do you see on the console? We have yet another surprise! The console shows we have an error in our Copy-Item. But why? It worked before when we ran it outside of a JOB.

I took a SWAG at it and decided to try enclosing the entire original script inside the JOB Scriptblock:

Here is the entire, final, working script:

$job = Start-Job -ScriptBlock {
    $source = "\\Lcfltr01\d$\SQL Backups\lcfltr01"
    $target = "W:\WoodyzTest"
    Copy-Item -Path $source -Destination $target -Recurse -Verbose
}
Wait-Job $job
Receive-Job $job