The Road to Cross-Platform Setup & Bootstrapping in .NET

TL;DR: PowerShell and Bash scripts are indispensable for developers. While benefiting from being transparent compared to executables, they’re not exactly easy to read by users nor easy to write by maintainers, and also suffer from duplication for different operating systems. With the .NET Core SDK, this issue is mostly solved using .NET Global Tools. Nevertheless, global tools come at the cost of requiring the SDK to be pre-installed. Several workarounds for local and build server environments exist, however, shell scripts are really the only solid solution. Single-entry scripts can help to erase the last relics from PowerShell and Bash scripts for documentation and end-users.

When I started with NUKE, part of the hardest work was to provide a convenient setup experience. NUKE was one of the first build systems to use conventional console applications in order to support first-class development experience. BullsEye and FlubuCore followed shortly after. Noteworthy, they come in a more simple form where the build project is isolated from the main solution by default. In contrast but not necessarily, NUKE allows us to add the build project to the solution besides other projects to make it more accessible. This approach works really great, but it does require some tedious tweaks that are good to automate. Another natural consequence of using build projects, is that they require a .NET Core SDK installation to build and run, which is crucial for build servers and new project contributors.

In the next sections, we’ll look at how the setup experience in NUKE evolved from complex PowerShell and Bash scripts to using .NET Global Tools, and how the .NET Core bootstrapping via shell scripts makes everything more cross-platform and self-contained.

Hacking Shell Scripts

Back in 2017, in the times of netstandard1.6, .NET Core was still on its journey of getting widely adopted in the .NET ecosystem. It was also the time when I made my first steps with JetBrains Rider, and moved to macOS for the major part of my development. But of course, many developers still relied on the .NET Framework, so for me it was crucial to make NUKE’s setup process to support both platforms. Nobody likes to download and run arbitrary executables from the web, especially when the projet is barely known. Therefore, I went with the more transparent PowerShell and Bash scripts, which can be easily investigated before execution:

1
2
3
4
5
6
powershell -Command iwr https://nuke.build/powershell -OutFile setup.ps1
powershell -ExecutionPolicy ByPass -File .\setup.ps1

curl -Lsfo setup.sh https://nuke.build/bash
chmod +x setup.s
./setup.sh

Writing the PowerShell setup.ps1 script was mostly effortless, but the Bash setup.sh script had some really horrible code in it. For instance, downloading a file, replacing placeholders with actual values and writing everything to disk:

1
2
3
4
5
6
7
8
sed -e 's~_TARGET_FRAMEWORK_~'"$TARGET_FRAMEWORK"'~g' \
    -e 's~_BUILD_PROJECT_GUID_~'"$PROJECT_GUID"'~g' \
    -e 's~_BUILD_PROJECT_NAME_~'"$BUILD_PROJECT_NAME"'~g' \
    -e 's~_SOLUTION_DIRECTORY_~'"${SOLUTION_DIRECTORY_RELATIVE//\//\\}"'~g' \
    -e 's~_NUKE_VERSION_~'"$NUKE_VERSION"'~g' \
    -e 's~_NUKE_VERSION_MAJOR_MINOR_~'"${NUKE_VERSION_PARTS[0]}.${NUKE_VERSION_PARTS[1]}"'~g' \
    <<<"$(curl -Lsf $BOOTSTRAPPING_URL/.build.$PROJECT_FORMAT.csproj)" \
    > "$BUILD_PROJECT_FILE"

Another example, that scans an existing file for specific lines, and adds additional lines after them:

1
2
awk "/MinimumVisualStudioVersion/{print \$0 RS \"$PROJECT_DEFINITION\";next}1" "$SOLUTION_FILE" > "$SOLUTION_FILE.bak"
awk "/ProjectConfigurationPlatforms/{print \$0 RS \"$PROJECT_CONFIGURATION\";next}1" "$SOLUTION_FILE.bak" > "$SOLUTION_FILE"

Using the Bash support and PowerShell support plugins for JetBrains Rider, made it a lot more bearable to work with those languages that I’m generally not fluent with. Especially, when it comes to syntax highlighting, code completion, navigation, renaming variables, fixing minor code issues, or quickly adding comments:

Even though the plugins helped a lot, honestly, I can’t say that I really enjoyed maintaining the setup scripts. I’ve never felt familiar with the code after coming back, and there were plenty of stupid and surprising issues, like missing parentheses or crashes if IE wasn’t started once before.

Setup with Global Tools

With netcoreapp2.1 the .NET Core SDK introduced .NET Global Tools and since NUKE meanwhile established a reasonable community, I’ve decided that it’s okay moving towards a unified executable:

Global tools are built from a single source and can be installed and executed on Windows, Linux and macOS. So no matter what the environment was, the setup experience for NUKE became as easy as:

1
2
dotnet tool install Nuke.GlobalTool --global
nuke :setup

That day the shell scripts have been replaced by a global tool, I could delete 400 LOC that were basically duplicated and hard to maintain, and replace them with roughly 300 lines of clean C# code.

Bootstrapping .NET Core SDK

Another difference compared to BullsEye and FlubuCore, is that NUKE provides bootstrapping scripts that will install the .NET Core SDK without any further requirements. As an example, here is the build.sh script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
#!/usr/bin/env bash

bash --version 2>&1 | head -n 1

set -eo pipefail
SCRIPT_DIR=$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)

###########################################################################
# CONFIGURATION
###########################################################################

BUILD_PROJECT_FILE="$SCRIPT_DIR/_BUILD_DIRECTORY_/_BUILD_PROJECT_NAME_.csproj"
TEMP_DIRECTORY="$SCRIPT_DIR/_ROOT_DIRECTORY_/.tmp"

DOTNET_GLOBAL_FILE="$SCRIPT_DIR/_ROOT_DIRECTORY_/global.json"
DOTNET_INSTALL_URL="https://dot.net/v1/dotnet-install.sh"
DOTNET_CHANNEL="Current"

export DOTNET_CLI_TELEMETRY_OPTOUT=1
export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
export DOTNET_MULTILEVEL_LOOKUP=0

###########################################################################
# EXECUTION
###########################################################################

function FirstJsonValue {
    perl -nle 'print $1 if m{"'"$1"'": "([^"]+)",?}' <<< "${@:2}"
}

# If dotnet CLI is installed globally and it matches requested version, use for execution
if [ -x "$(command -v dotnet)" ] && dotnet --version &>/dev/null; then
    export DOTNET_EXE="$(command -v dotnet)"
else
    # Download install script
    DOTNET_INSTALL_FILE="$TEMP_DIRECTORY/dotnet-install.sh"
    mkdir -p "$TEMP_DIRECTORY"
    curl -Lsfo "$DOTNET_INSTALL_FILE" "$DOTNET_INSTALL_URL"
    chmod +x "$DOTNET_INSTALL_FILE"

    # If global.json exists, load expected version
    if [[ -f "$DOTNET_GLOBAL_FILE" ]]; then
        DOTNET_VERSION=$(FirstJsonValue "version" "$(cat "$DOTNET_GLOBAL_FILE")")
        if [[ "$DOTNET_VERSION" == ""  ]]; then
            unset DOTNET_VERSION
        fi
    fi

    # Install by channel or version
    DOTNET_DIRECTORY="$TEMP_DIRECTORY/dotnet-unix"
    if [[ -z ${DOTNET_VERSION+x} ]]; then
        "$DOTNET_INSTALL_FILE" --install-dir "$DOTNET_DIRECTORY" --channel "$DOTNET_CHANNEL" --no-path
    else
        "$DOTNET_INSTALL_FILE" --install-dir "$DOTNET_DIRECTORY" --version "$DOTNET_VERSION" --no-path
    fi
    export DOTNET_EXE="$DOTNET_DIRECTORY/dotnet"
fi

echo "Microsoft (R) .NET Core SDK version $("$DOTNET_EXE" --version)"

"$DOTNET_EXE" build "$BUILD_PROJECT_FILE" /nodeReuse:false /p:UseSharedCompilation=false -nologo -clp:NoSummary --verbosity quiet
"$DOTNET_EXE" run --project "$BUILD_PROJECT_FILE" --no-build -- "$@"

The bootstrapping scripts are very important to run on build servers, since the required .NET Core SDK might not be installed:

Now, many will say that there is the Use .NET Core task that will perform an install according to the global.json file. In practice, this approach has several disadvantages. Firstly, a similar task might not exist on the particular build server we’re using. Also, predefined tasks tend to be very inflexible and eventually force us to use hacky workarounds or reconsider going back to shell scripts:

Even more importantly, with the rapid evolution of C# as a language, chances are good that we might not have the required SDK installed for a repository we’ve just forked. Joseph Woodward came up with the InstallSdkGlobalTool to partially solve that for local machines at least. So in a repository with global.json file, we can just invoke dotnet-install-sdk and it will do its magic:

Remember that this requires at least some .NET Core SDK to be installed. Also keep in mind that this is a manual step and the tool needs to be installed on our colleagues or contributors machine.

Single-Entry Scripts

So far, the experience was looking quite good already. But one thing that always bothered me, was that build.sh and build.ps1 were exclusive to UNIX and Windows systems. Can’t we have a single-entry script that can be invoked cross-platform? That would solve the following issues:

• Documentation – Duplicated invocation snippets for both build.sh and build.ps1 are a waste of time while reading, annoying when writing documentation, and also more prone to becoming outdated.
• Multi-platform pipelines – Many build servers enable us to write multi-platform pipelines where a set of common build steps is executed for different operating systems using a matrix approach. This works great with pre-defined tasks, but it falls short when we need to call different scripts.

Based on a Stack Overflow question, I’ve found that the : colon character can be used to effectively skip code in Batch scripts, while POSIX shells simply evaluate them to true:

1
2
3
4
5
6
7
:; set -eo pipefail
:; SCRIPT_DIR=$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)
:; ${SCRIPT_DIR}/build.sh "$@"
:; exit $?

@ECHO OFF
powershell -ExecutionPolicy ByPass -NoProfile %0\..\build.ps1 %*

Depending on our current operating system, calling build.cmd will either delegate to build.sh or build.ps1. It’s probably obvious, that for larger and more complex scripts, this approach doesn’t scale very well. Most of the IDE tooling won’t work apart from very simple hippie completion.

After some time Georg Dangl discovered an issue that the generated scripts are not executable by default. When invoking the scripts, the error message permission denied: ./build.cmd was returned. Meanwhile, this has been addressed so that the setup in NUKE now executes the following commands:

• chmod +x build.cmd – Allow local execution
• git update-index --add --chmod=+x build.cmd – Mark file to be executable in Git
• svn propset svn:executable on build.cmd – Mark file to be executable in Subversion

Conclusion

With .NET Core running cross-platform, it is important to ensure shell scripts to work cross-platform as well. Setup routines are perfectly qualified to be moved into global tools, whereas bootstrapping tasks, like installing the .NET Core SDK, should remain in PowerShell and Bash scripts to improve the experience for build servers and first-time contributors.

Bootstrap your builds frictionless with NUKE!