Share

Blog minification in a fancy way with PowerShell

There is no built-in support for minification in Hugo engine, which isn’t a bad thing on its own. But I had to figure out a way to minify my files while keeping building and maintenance extremely simple.

At first point I added minify for text files. It’s a single binary (written in Go), and with a single call it can run recursively over Hugo’s public folder, minifying output files in-place, just as I needed.

But minifying images is a more tricky task. I decided to use jpegtran and optipng, but they can’t be run recursively. Some scripting is required to use them. While the task of finding all the images in a folder and applying minifiers on them is trivial and can be solved with PowerShell one-liner, it was not satisfying enough, so I end up with the thing described below in this post.

Additional motivation is to have a tool that can be adopted easily to any other processing task and will give me a fancy progress report.


What I need to do

Initially I have the output folder where Hugo engine puts everything in a way it should be hosted on a web server — bunch of htmls, styles, script, images and whatever else. I need to run recursively through all the files and apply corresponding minifying tool on each of them.

Running through all the text files with minify is quite fast, but png optimization takes some time. With the amount of files growing with the blog, it will be more and more time consuming.

So the first task is to make the process measurable, providing meaningful information at any point of time. We can show: overall progress, time estimation, information about files processed.

Steps of implementation:

  1. make a single file processing function;
  2. make a pipeline for all files to be processed;
  3. tune everything.

It’s all quite obvious, so I’ll just point out some details below and provide complete script.

All the basic PowerShell stuff used

(provided for the sake of completeness)

Considerations for table output

Some measures have been taken in order to minimize clutter on a screen by proper information organization. All columns are given proper widths. Rows are grouped by folders, so irrelevant part of folder paths is stripped by using Resolve-Path -Relative. This script meant to be run from the same folder as Hugo itself, and I prefer to see that the script is running inside public directory (all relative paths will start from “./public/”). In general case, more complex logic may be needed to make relative paths from one place to another — either by changing current directory back and forth or by using System.Uri class.

I thought I will need a horizontal line after processed files and before total numbers. So I looked for a way to draw it perfectly to the size of terminal window. Get-Host command will give you the information about terminal. Usually across examples in the internet, $(get-host).UI.RawUI.WindowSize is used. I had some issues with that when run my PowerShell script from bat file in ConsoleZ — I think it was some rare bug (can’t reproduce), but I decided to stick with $(get-host).UI.RawUI.BufferSize since then. And keep in mind that usable width is one character less than returned buffer/window width.

This script was written in Windows 8.1 with PowerShell v4. After upgrading to Windows 10 I found that Format-Table behaviour was changed in PowerShell v5. Now it produces “compactified” tables instead of full-width tables. And there is no switch that will give you the old behaviour. It’s OK in many cases, but NOT when you’re listing files and do not want to hold pipeline output. Width of first column with file path varies a lot, and now, in order to reserve all the available space for it, you have to do all the math with buffer width. When doing math, take into account that there is additional space between columns.

Considerations for progress bar

In order to use Write-Progress cmdlet inside pipeline I’ve taken a function from there and updated it a little bit to my taste. $Input automatic variable is the most interesting thing here. $Input.Count is checked before processing — it will be an issue later, but not today.

I’ve also added a very crude remaining time estimation. Interesting point here is that progress bar allows only for three lines of text to be set by user — Activity, CurrentOperation and Status. In order to avoid clutter in Status line, I’ve repurposed CurrentOperation line for estimated time. There is no multiplication operation defined for TimeSpan, so additional function was introduced just for that.

Another gotcha is the way how to call a pipeline function with additional arguments: you set them by name, like this: ... | Show-Progress -Activity "Minifying files" | .... It’s a PowerShell thing you need to wrap your head around, and, surprisingly, I found no pipeline function examples with additional arguments on the Internet. Too basic stuff to be bothered mentioning, as it seems…

Complete script


#region helpers

function Multiply-Timespan([TimeSpan]$timespan, [double]$multiplier) {
    [long]$ticks = $timespan.Ticks * $multiplier
    return [TimeSpan]::FromTicks($ticks)
}

function Show-Progress{
    [CmdletBinding()]
    param (
        [Parameter(Mandatory=$true, Position=0, ValueFromPipeline=$true)]
        [PSObject[]]$InputObject,
        [string]$Activity = "Processing items"
    )

    [int]$TotalItems = $Input.Count
    [int]$Count = 0
    $startTime = Get-Date

    $Input | foreach {
        $_
        $Count++
        $partComplete = 1.0 * $Count / $TotalItems
        $now = Get-Date
        $remaining = Multiply-Timespan ($now - $startTime) ((1 - $partComplete) / $partComplete)
        $status = "{0} / {1} ( {2:P1} )" -f ($Count, $TotalItems, $partComplete, $remaining)
        $moreInfo = "remaining time: {0:hh\:mm\:ss}" -f $remaining
        Write-Progress -Activity $Activity `
                       -PercentComplete (100 * $partComplete) `
                       -Status ($status) `
                       -CurrentOperation ($moreInfo)
    }
    
    Write-Progress -Activity $Activity -Completed
}

function Draw-Line($char = "="){
    $char * ($(get-host).UI.RawUI.BufferSize.Width - 1)
}

#endregion


#region minification

$jpegExtensions = ".jpg", ".jpeg"
$pngExtensions = ,".png"
$textExtensions = ".css", ".htm", ".html", ".js", ".json", ".svg", ".xml"
$allExtensions = $jpegExtensions + $pngExtensions + $textExtensions

function Optimize-File([System.IO.FileInfo]$InputFile) {
    $path = $InputFile.FullName
    $ext = $InputFile.Extension
    $fromSize = $InputFile.length
    if( $ext -in $jpegExtensions )
    {
        & jpegtran.exe -copy none -optimize -progressive -outfile "$path" "$path"
    }
    if( $ext -in $pngExtensions )
    {
        & optipng.exe -quiet -o3 -clobber "$path"
    }
    if( $ext -in $textExtensions )
    {
        & minify.exe --html-keep-whitespace -o "$path" "$path"
    }
    $toSize = (Get-Item $path).length
    return [PSCustomObject]@{
        Name=$InputFile.Name;
        From=$fromSize;
        To=$toSize;
        Folder=($InputFile.Directory | Resolve-Path -Relative);
        Ext=$ext
        }
}

$colW = 12
$firstColW = $(get-host).UI.RawUI.BufferSize.Width - 1 - ($colW+1)*4

Get-ChildItem -Path .\public -Recurse -File |
    Where Extension -in $allExtensions |
    Show-Progress -Activity "Minifying files" |
    %{ Optimize-File $_ } |
    tee -Variable results |
    Format-Table -GroupBy Folder `
                 -Property @{ Label="Name"; Width=$firstColW; Expression={$_.Name} },
                           @{ Label="From"; Width=$colW; Expression={$_.From} },
                           @{ Label="To"; Width=$colW; Expression={$_.To} },
                           @{ Label="Saved"; Alignment="Right"; Width=$colW; `
                              Expression={"{0}" -f ($_.From - $_.To)} },
                           @{ Label="Percentage"; Alignment="Right"; Width=$colW; `
                              Expression={"{0:P1}" -f ($_.To/$_.From)} }

Draw-Line
""
"   Totals"

$results |
    Group-Object { $_.Ext } |
    Select-Object -Property Count,
                            @{ Name = "Extension"; Expression = {$_.Name} },
                            @{ Name = "From"; `
                               Expression = {($_.Group | Measure-Object -Property From -Sum).Sum} },
                            @{ Name = "To"; `
                               Expression = {($_.Group | Measure-Object -Property To -Sum).Sum} } |
    Format-Table -Property @{ Label="Extension"; Width=$firstColW; Expression={$_.Extension} },
                           @{ Label="From"; Width=$colW; Expression={$_.From} },
                           @{ Label="To"; Width=$colW; Expression={$_.To} },
                           @{ Label="Saved"; Alignment="Right"; Width=$colW; `
                              Expression={"{0}" -f ($_.From - $_.To)} },
                           @{ Label="Percentage"; Alignment="Right"; Width=$colW; `
                              Expression={"{0:P1}" -f ($_.To/$_.From)} }

#endregion

It produces output like this:

progress bar

... (skipped most part) ...

   Folder: .\public\project\hugo-site

Name                                                 From           To        Saved   Percentage
----                                                 ----           --        -----   ----------
index.html                                          12425         8916         3509        71,8%


   Folder: .\public\project\page\1

Name                                                 From           To        Saved   Percentage
----                                                 ----           --        -----   ----------
index.html                                            241          157           84        65,1%


   Folder: .\public\project\paintdotnetstuff

Name                                                 From           To        Saved   Percentage
----                                                 ----           --        -----   ----------
dimension.png                                        1518         1172          346        77,2%
index.html                                          17305        13371         3934        77,3%
shapes_misc_grids_sample.png                        64776        57287         7489        88,4%
shapes_misc_grids_sample_thumb.png                  23174        21762         1412        93,9%
shapes_photo_sample.png                             25418        19428         5990        76,4%
shapes_photo_sample_thumb.png                        6717         5386         1331        80,2%
shapes_rulers_and_measurements_sample.png           26046        22051         3995        84,7%
shapes_rulers_and_measurements_sample_thumb.png     11036         9379         1657        85,0%
shapes_superellipse_sample.png                      32430        28573         3857        88,1%
shapes_superellipse_sample_thumb.png                 9632         8218         1414        85,3%


================================================================================================

   Totals

Extension                                            From           To        Saved   Percentage
---------                                            ----           --        -----   ----------
.html                                              327114       247261        79853        75,6%
.xml                                               222109       213591         8518        96,2%
.png                                               422439       351662        70777        83,2%
.css                                                43078        39482         3596        91,7%
.svg                                               392766       362160        30606        92,2%
.js                                                118030       117276          754        99,4%
.jpg                                              2636027      2485277       150750        94,3%

Website benchmarks like WebPagetest and GTmetrix are mostly happy now.

What’s next

  1. Make it work — check.
  2. Make it right — …check (constructive feedback is welcomed).
  3. Make it fast — i leave it for the next post.

Stay tuned.

comments powered by Disqus