Contents:
There is no built-in support for minification in Hugo engine, which isn’t a bad thing on its own. But I had to figure out a way to minify my files while keeping building and maintenance extremely simple.
At first point I added minify for text files. It’s a single binary (written in Go), and with a single call it can run recursively over Hugo’s public folder, minifying output files in-place, just as I needed.
But minifying images is a more tricky task. I decided to use jpegtran and optipng, but they can’t be run recursively. Some scripting is required to use them. While the task of finding all the images in a folder and applying minifiers on them is trivial and can be solved with PowerShell one-liner, it was not satisfying enough, so I end up with the thing described below in this post.
Additional motivation is to have a tool that can be adopted easily to any other processing task and will give me a fancy progress report.
What I need to do
Initially I have the output folder where Hugo engine puts everything in a way it should be hosted on a web server — bunch of htmls, styles, script, images and whatever else. I need to run recursively through all the files and apply corresponding minifying tool on each of them.
Running through all the text files with minify is quite fast, but png optimization takes some time. With the amount of files growing with the blog, it will be more and more time consuming.
So the first task is to make the process measurable, providing meaningful information at any point of time. We can show: overall progress, time estimation, information about files processed.
Steps of implementation:
- make a single file processing function;
- make a pipeline for all files to be processed;
- tune everything.
It’s all quite obvious, so I’ll just point out some details below and provide complete script.
All the basic PowerShell stuff used
(provided for the sake of completeness)
- Call operator
&to run commands; - Get-ChildItem to list files recursively;
- pipelines and ForEach-Object
%for, well, processing pipeline; - format operator
-fto format strings (you can use $variables right inside strings when it’s more convenient); - comparison operators — inclusion
-inoperator is extremely handy; [PSCustomObject]@{}to return all relevant information from processing function for reporting needs (see this post for some details);- tee object to keep results from pipeline for later use — very handy when you need to calculate totals;
- Format-Table with calculated properties and customized columns;
- Group-Object, Select-Object and Measure-Object to calculate totals;
- Write-Progress to show the progress of long operations.
Considerations for table output
Some measures have been taken in order to minimize clutter on a screen by proper information organization. All columns are given proper widths. Rows are grouped by folders, so irrelevant part of folder paths is stripped by using Resolve-Path -Relative. This script meant to be run from the same folder as Hugo itself, and I prefer to see that the script is running inside public directory (all relative paths will start from “./public/”). In general case, more complex logic may be needed to make relative paths from one place to another — either by changing current directory back and forth or by using System.Uri class.
I thought I will need a horizontal line after processed files and before total numbers. So I looked for a way to draw it perfectly to the size of terminal window. Get-Host command will give you the information about terminal. Usually across examples in the internet, $(get-host).UI.RawUI.WindowSize is used. I had some issues with that when run my PowerShell script from bat file in ConsoleZ — I think it was some rare bug (can’t reproduce), but I decided to stick with $(get-host).UI.RawUI.BufferSize since then. And keep in mind that usable width is one character less than returned buffer/window width.
This script was written in Windows 8.1 with PowerShell v4. After upgrading to Windows 10 I found that Format-Table behaviour was changed in PowerShell v5. Now it produces “compactified” tables instead of full-width tables. And there is no switch that will give you the old behaviour. It’s OK in many cases, but NOT when you’re listing files and do not want to hold pipeline output. Width of first column with file path varies a lot, and now, in order to reserve all the available space for it, you have to do all the math with buffer width. When doing math, take into account that there is additional space between columns.
Considerations for progress bar
In order to use Write-Progress cmdlet inside pipeline I’ve taken a function from there and updated it a little bit to my taste. $Input automatic variable is the most interesting thing here. $Input.Count is checked before processing — it will be an issue later, but not today.
I’ve also added a very crude remaining time estimation. Interesting point here is that progress bar allows only for three lines of text to be set by user — Activity, CurrentOperation and Status. In order to avoid clutter in Status line, I’ve repurposed CurrentOperation line for estimated time. There is no multiplication operation defined for TimeSpan, so additional function was introduced just for that.
Another gotcha is the way how to call a pipeline function with additional arguments: you set them by name, like this: ... | Show-Progress -Activity "Minifying files" | .... It’s a PowerShell thing you need to wrap your head around, and, surprisingly, I found no pipeline function examples with additional arguments on the Internet. Too basic stuff to be bothered mentioning, as it seems…
Complete script
#region helpers
function Multiply-Timespan([TimeSpan]$timespan, [double]$multiplier) {
[long]$ticks = $timespan.Ticks * $multiplier
return [TimeSpan]::FromTicks($ticks)
}
function Show-Progress{
[CmdletBinding()]
param (
[Parameter(Mandatory=$true, Position=0, ValueFromPipeline=$true)]
[PSObject[]]$InputObject,
[string]$Activity = "Processing items"
)
[int]$TotalItems = $Input.Count
[int]$Count = 0
$startTime = Get-Date
$Input | foreach {
$_
$Count++
$partComplete = 1.0 * $Count / $TotalItems
$now = Get-Date
$remaining = Multiply-Timespan ($now - $startTime) ((1 - $partComplete) / $partComplete)
$status = "{0} / {1} ( {2:P1} )" -f ($Count, $TotalItems, $partComplete, $remaining)
$moreInfo = "remaining time: {0:hh\:mm\:ss}" -f $remaining
Write-Progress -Activity $Activity `
-PercentComplete (100 * $partComplete) `
-Status ($status) `
-CurrentOperation ($moreInfo)
}
Write-Progress -Activity $Activity -Completed
}
function Draw-Line($char = "="){
$char * ($(get-host).UI.RawUI.BufferSize.Width - 1)
}
#endregion
#region minification
$jpegExtensions = ".jpg", ".jpeg"
$pngExtensions = ,".png"
$textExtensions = ".css", ".htm", ".html", ".js", ".json", ".svg", ".xml"
$allExtensions = $jpegExtensions + $pngExtensions + $textExtensions
function Optimize-File([System.IO.FileInfo]$InputFile) {
$path = $InputFile.FullName
$ext = $InputFile.Extension
$fromSize = $InputFile.length
if( $ext -in $jpegExtensions )
{
& jpegtran.exe -copy none -optimize -progressive -outfile "$path" "$path"
}
if( $ext -in $pngExtensions )
{
& optipng.exe -quiet -o3 -clobber "$path"
}
if( $ext -in $textExtensions )
{
& minify.exe --html-keep-whitespace -o "$path" "$path"
}
$toSize = (Get-Item $path).length
return [PSCustomObject]@{
Name=$InputFile.Name;
From=$fromSize;
To=$toSize;
Folder=($InputFile.Directory | Resolve-Path -Relative);
Ext=$ext
}
}
$colW = 12
$firstColW = $(get-host).UI.RawUI.BufferSize.Width - 1 - ($colW+1)*4
Get-ChildItem -Path .\public -Recurse -File |
Where Extension -in $allExtensions |
Show-Progress -Activity "Minifying files" |
%{ Optimize-File $_ } |
tee -Variable results |
Format-Table -GroupBy Folder `
-Property @{ Label="Name"; Width=$firstColW; Expression={$_.Name} },
@{ Label="From"; Width=$colW; Expression={$_.From} },
@{ Label="To"; Width=$colW; Expression={$_.To} },
@{ Label="Saved"; Alignment="Right"; Width=$colW; `
Expression={"{0}" -f ($_.From - $_.To)} },
@{ Label="Percentage"; Alignment="Right"; Width=$colW; `
Expression={"{0:P1}" -f ($_.To/$_.From)} }
Draw-Line
""
" Totals"
$results |
Group-Object { $_.Ext } |
Select-Object -Property Count,
@{ Name = "Extension"; Expression = {$_.Name} },
@{ Name = "From"; `
Expression = {($_.Group | Measure-Object -Property From -Sum).Sum} },
@{ Name = "To"; `
Expression = {($_.Group | Measure-Object -Property To -Sum).Sum} } |
Format-Table -Property @{ Label="Extension"; Width=$firstColW; Expression={$_.Extension} },
@{ Label="From"; Width=$colW; Expression={$_.From} },
@{ Label="To"; Width=$colW; Expression={$_.To} },
@{ Label="Saved"; Alignment="Right"; Width=$colW; `
Expression={"{0}" -f ($_.From - $_.To)} },
@{ Label="Percentage"; Alignment="Right"; Width=$colW; `
Expression={"{0:P1}" -f ($_.To/$_.From)} }
#endregion
It produces output like this:
... (skipped most part) ...
Folder: .\public\project\hugo-site
Name From To Saved Percentage
---- ---- -- ----- ----------
index.html 12425 8916 3509 71,8%
Folder: .\public\project\page\1
Name From To Saved Percentage
---- ---- -- ----- ----------
index.html 241 157 84 65,1%
Folder: .\public\project\paintdotnetstuff
Name From To Saved Percentage
---- ---- -- ----- ----------
dimension.png 1518 1172 346 77,2%
index.html 17305 13371 3934 77,3%
shapes_misc_grids_sample.png 64776 57287 7489 88,4%
shapes_misc_grids_sample_thumb.png 23174 21762 1412 93,9%
shapes_photo_sample.png 25418 19428 5990 76,4%
shapes_photo_sample_thumb.png 6717 5386 1331 80,2%
shapes_rulers_and_measurements_sample.png 26046 22051 3995 84,7%
shapes_rulers_and_measurements_sample_thumb.png 11036 9379 1657 85,0%
shapes_superellipse_sample.png 32430 28573 3857 88,1%
shapes_superellipse_sample_thumb.png 9632 8218 1414 85,3%
================================================================================================
Totals
Extension From To Saved Percentage
--------- ---- -- ----- ----------
.html 327114 247261 79853 75,6%
.xml 222109 213591 8518 96,2%
.png 422439 351662 70777 83,2%
.css 43078 39482 3596 91,7%
.svg 392766 362160 30606 92,2%
.js 118030 117276 754 99,4%
.jpg 2636027 2485277 150750 94,3%
Website benchmarks like WebPagetest and GTmetrix are mostly happy now.
What’s next
- Make it work — check.
- Make it right — …check (constructive feedback is welcomed).
- Make it fast — I leave it for the next post.
Stay tuned.
