Running arbitrary number of tasks in parallel with PowerShell, while showing overall progress.

The main disadvantage of the code I provided in previous post is that it is performed sequentially. We can speed it up by means of parallel processing, since main limiting factor is not an I/O but minifying algorithms performance.

There are a lot of ways to use parallelism in PowerShell and it wasn’t clear for me which one suits well for my task. I.e. to convert my script, which is processing a lot of small files on a local machine.

It also should:

• provide updates in real time;
• keep output table formatting as close to original as possible.

---

Some changes to prepare for parallel execution

With parallel execution, items order is no longer guaranteed, so grouping them by folder makes no more sense, and it’s no longer possible to save screen space by that. Which means I need to put file path into one line. When formatting table, PowerShell will just trim it at the end, which isn’t very useful, so I’ve implemented my own ellipsis function to deal with it. I’m dot-sourcing that to keep everything in one place.

Version of progress bar function that I used in previous post is not really suitable for parallel execution. I upgraded it in a way that will not hold the pipeline, so it can be put after processing code and counting completed (rather than started) items in real time. One disadvantage of a new function is that it still requires a total amount of items and that should be passed explicitly now.

And I moved helper functions to separate file since I will call them from various places:

Slightly updated version of previous code is available here.

ForEach Parallel, Workflows

My first approach was to use ForEach -Parallel. The idea of transforming ordinary ForEach loop into parallel code seemed very attractive. But some significant disadvantages showed themselves in process of adopting it to my task. It can only be used inside of workflows (more) and you should call your code with Invoke-Command from within InlineScripts.

One tricky moment is that ScriptBlocks cannot be passed into InlineScripts as is, so they’re casted to strings and should be casted back inside. Another quirk is that you need to keep track of current location, as InlineScript started at default location. Read more about workflow restrictions.

There is also a hardcoded progress bar for InlineScript that is working in undesired way — it can be suppressed with $progressPreference = 'SilentlyContinue'.

And at the end, not only it turned to be really dodgy, but achieved performance was absolutely unsatisfactory — it was even slower than sequential code in my case.

My code is here:

Background Jobs

Background job is a tool to perform actions asynchronously and separate from main thread. There is an example how to run them in a limited number of threads, so I’ve adopted it to my case.

With all the cleaning-up routine and waiting timers it doesn’t look pretty, and it also turned out to be really slow. Another tool not fit for the task.

My code is here:

Runspaces

The next candidate was the Runspace class.

Based on detailed answer from there and another example there I was able to write working code, although it was even less pretty than one with background jobs. More about runspaces there.

I’ve got some issues with function chaining, so there are a lot of Out-Nulls inside. But other than that it works fine and it is a thing I should’ve started from.

My code is here:

Then I checked what kinds of 3rd party code available for the task:

InvokeParallel

InvokeParallel by RamblingCookieMonster is a wrapper for runspaces.

As a wrapper, it removes the need to handle runspaces by myself and provides simpler interface with additional features. It is possible to pass variables into paralleled code (but not functions, at least not now). And it shows a progress bar in a way close to what I need.

Unfortunately, it cannot handle functions on input — only actual scriptblocks are accepted, not @{function:foo}. But other than that, it fits well for my task and produces cleanest code out of all options listed here.

It’s quite surprizing that this thing is not distributed as a module — you can only get it from GitHub and dot-source the .ps1 file.

My code is here:

PoshRSJob

PoshRSJob from proxb is another wrapper for runspaces. It is available as a module, and can be installed with Install-Module since PowerShell v5.

This module provides a set of functions similar to ones for background jobs, so I started from adopting my background jobs code to this module. It was a wrong way to go and I’ve spent some time facing some crazy issues before I threw it out and switched to simple example with Start—Wait—Receive pipeline. Finally, it was working fine and with short and clean code.

After all my issues I’ve got a feeling that this module is only reliable when used in one particular way. I hope this is just my mistake. And it also got a major update recently.

PoshRSJob, like InvokeParallel, is not accepting @{function:foo} as scriptblock. Variables are accessed with $using:bar syntax. There is no built-in progress bar feature. But it has a unique feature of importing functions (which got no use in my example, unfortunately).

My code is here:

Time measurements

Test setup:

• my blog as a source of content for minification;
• between tests 1, 2 and 3 there was varying amount of content provided for minification;
• new PowerShell session opened every time, as I’m trying to test cold run performance.

Provided values are an average number of seconds from 3 runs and relative standard deviation (standard deviation over average).

Time as reported by psake for minifying task alone, after command invoke-psake -taskList Build, ... completed with exact task name (all the tasks are listed in default.ps1).

In case of tasks with sorting, small files are left for the end of the queue — I’m trying to shave a second in the end, ensuring that there will be no long tasks, and checking an actual effect. For sequential implementation it makes no sense.

Option	Test 1 (sec)	RSD	Test 2 (sec)	RSD	Test 3 (sec)	RSD
Sequential	12,79	0,3%	21,10	1,4%	38,61	0,5%
InvokeParallel	7,61	1,2%	9,55	3,2%	13,58	0,5%
InvokeParallel w/ sorting	6,56	2,2%	8,84	0,7%	13,38	1,8%
PoshRSJob	7,45	2,7%	9,16	2,6%	12,80	1,0%
PoshRSJob w/ sorting	6,66	1,3%	8,54	2,9%	12,49	0,4%
Runspace	5,28	1,6%	7,00	0,4%	10,69	1,5%
Runspace w/ sorting	4,50	0,5%	6,29	2,0%	10,26	0,8%
Workflow	24,13	0,1%	29,60	1,4%	43,92	1,2%
Workflow w/ sorting	23,79	0,5%	29,64	2,1%	42,72	0,3%
Job	31,51	2,4%	37,61	0,7%	58,11	0,4%
Job w/ sorting	30,73	3,4%	38,09	0,6%	58,24	1,5%

What observations can be made from the results:

• Workflows with ForEach Parallel and Jobs (Start-Job, etc) are completely wrong tools for this task.

• Runspaces are your friends when you need to run things as fast as possible in PowerShell.

• InvokeParallel and PoshRSJob are wrappers over runspaces with some added value and some added overhead too. They are quite similar to each other in terms of performance, with PoshRSJob being slightly faster.

• Sorting files queue prevents the situation when there is a long job in the end that cannot be paralleled, but this effect is not scalable and becoming less and less important with the growing amount of files.

Following plot (made with plot.ly) gives more insight:

• Sequential numbers are on a straight line at 45 degree — it’s a reference. Everything above is slower and everything below is faster;

• PoshRSJob stays in parallel with Runspace, which means it has only constant overhead;

• InvokeParallel goes up a bit with the amount of content, which means it has some additional overhead per item;

• much bigger numbers required to see where Workflows will became more efficient than sequential run;

• background jobs are behaving strangely. I checked the code: multiple threads are running, indeed, but after Receive-Job it worth to call Remove-Job too. It changes the running time somewhat, but not dramatically. I’ve updated the gist and will update my measurements later.

Features

Option	@{function:foo}	Variables	Functions	Modules	Availability	LOC
	1	2	3	4	5	6
InvokeParallel	–	+	–	+	ps1 file	91
PoshRSJob	–	$using:	+	+	module	91
Runspace	+	–	–	–	PS v2	135
Workflow	+	–	–	–	PS v3	117
Job	+	–	–	–	PS v2	122

(1) ability to pass a function as a scriptblock to be executed in parallel;

(2,3,4) ability to pass variables, functions and modules from current scope into the code executed in parallel;

(5) what is required to use this type of code;

(6) number of lines of code doing the same thing in my examples.

Conclusion

There are a lot of suggestions in the Internet like “there are runspaces, but you should use them in a ready-made wrapper like InvokeParallel or PoshRSJob“. That’s right. But I think it is also not a bad idea to have much simpler wrapper (mine is just 40 lines long) in your sleeve. Depending on a task it can save you some runtime and reduce 3rd-party dependencies amount if you have no special feature requirements.

Workflows with ForEach Parallel and background Jobs are both seems to have huge overhead and not meant to be used for processing on a local machine. Not for short tasks like in this example at least.

As a result of my little survey, I’m providing a full set of working examples as a multifile gist. Go check it out (if you didn’t yet). Binary dependencies are listed in readme file.

And now I have an optimal blog building and minification routine in psake tasks (I’ll stick with Runspaces).