30th of Jan 2022: Many .NET open-source have now migrated to GitHub Actions. I hence decided to include it in this post.
25th of Sep 2021: I decided to remove Travis CI from this post. Travis CI recently poorly handled a security vulnerability and security is of paramount importance when it comes to build systems.
5th of Jan 2019: a lot has been happening since I initially wrote this post.
Azure DevOps released a free tier for open source projects, the
GitVersion contributors have been hard at work to take advantage of the latest features of
.NET Core. So many things have changed that I decided to update this post to reflect the current state of affairs (inclusion of
Azure DevOps, upgrade to
.NET Core 2.2, utilisation of
.NET Core global tools and removing the
Mono requirement on
As a developer I’m amazed by the number of free tools and services available. I wanted to create an end-to-end demo of a
CI/CD pipeline that would include:
- Trigger a build on commit
- Use semantic versioning
- Run tests
- Publish test results
- Create NuGet packages
- Publish the NuGet packages
- Create a GitHub release
For my purpose I wanted anonymous users to have access to a read-only view. AppVeyor and GitHub Actions are the most popular choices for
.NET open-source projects. CircleCI seems to have dropped in popularity. Azure DevOps released (and then unreleased) a free and unlimited plan for open source projects. As I had both CircleCI and Azure DevOps working, I decided to keep them. In this post I’ll be leveraging the four platforms so that I can highlight their pros and cons.
GitHub Actions all use YAML configuration files. This means that your build steps are living in the same space than your code and this presents several benefits:
- Any developer can modify the build
- The project is self-contained
- Developers don’t have to search where the build is located
- It doesn’t matter if something terrible happens to the build server
- Ability to run the build locally on some platforms
YAML files are:
Azure DevOps: azure-pipelines.yml
GitHub Actions: .github/workflows/build.yml
The project is useless. What is important is that it describes a real-life scenario:
- The solution contains two projects which will be packed as
Logicproject references a
NuGetpackage from nuget.org via a
dotnet packwill turn this into a package reference.
SuperLogicproject depends on
Logicand when packing, this project reference will be turned into a
NuGetpackage reference (handled out of the box by
- The projects target both
net461so they can also be used with the
- The resulting
NuGetpackages should contain
DLLs for both frameworks
- The resulting
- The projects reference a third project that should be embedded as a
DLLrather than being referenced as a
- This is not yet supported by the new tooling but can be achieved.
Pinning the version of
Cake guarantees you’ll be using the same version of
Cake on your machine and on the build servers. This is achieved by installing Cake as a .NET local tool.
As I’m releasing packages I decided to use semantic versioning.
Consider a version format of
Major.Minor.Patch). Bug fixes not affecting the API increment the patch version, backwards compatible API additions/changes increment the minor version, and backwards incompatible API changes increment the major version.
Semantic versioning allows the consumers of your binaries to assess the effort to upgrade to a newer version. Semantic versioning should not be used blindly for all kinds of projects. It makes a lot of sense for a
NuGet package but it doesn’t for a product or an
API for example.
.NET we use four properties to handle versioning:
AssemblyInformationalVersionto version assemblies
PackageVersionto version a
Versioning an assembly
AssemblyVersion: the only version the
CLRcares about (if you use strong named assemblies)
AssemblyVersioncan be defined as
<major-version>.<minor-version>.<build-number>.<revision>where each of the four segment is a
stringthat attaches additional version information to an assembly for informational purposes only. Corresponds to the product’s marketing literature, packaging, or product name
AssemblyInformationalVersion is well documented.
AssemblyFileVersion: intended to uniquely identify a build of the individual assembly
Developers tend to auto-increment this on each build. I prefer it linked to a
tag to be able to reproduce a build. I also use the same
AssemblyFileVersion (I’m a bad person I know).
PackageVersion: A specific package is always referred to using its package identifier and an exact version number
NuGet package versioning is described here.
I’ve implemented semantic versioning using GitVersion. I recommend using GitHub Flow when working on a simple package. In my experience Trunk Based Development tends to lead to lower code quality. Developers push early and often thinking they will fix it later but we all know than in software development later means never.
GitVersion produces an output that will allow you to handle even the trickiest situations:
In my case I’m using:
If you want to handle rebasing and
Pull Requests you’ll have to use a more complex versioning strategy (keep in mind that
GitHub advises against force-push in
As an aside
Cake allows you to set the
AppVeyor build number.
I do not set the
AppVeyor build number any more as
GitHub Actions is pushing the
NuGet packages and creating the tags and
Run the tests
CircleCI build is running on
Linux it doesn’t support testing against
net461. Luckily the framework can be specified using an argument:
Publish the test results
Again, the integration between
AppVeyor shines in this area as
Cake will automatically publish the test results for you (I wondered why I had duplicate test results until I RTFM).
AppVeyor displays all the tests but you must hover to see the framework used:
Failed tests come with a nice formatting and a
Azure DevOps has the most powerful tests results tab:
CircleCI has a few quirks when it comes to testing.
First it only supports the JUnit format so I had to use the JunitXml.TestLogger
NuGet package to be able to publish the test results. Then you must place your test results within a folder named after the test framework you are using if you want
CircleCI to identify your test framework.
When the tests run successfully
CirceCI will only display the slowest test (you need to navigate to Test Insights to see it):
I don’t understand the use case, I would prefer the list of tests and timing and the ability to sort them client-side.
The output for failed tests is much better when using a
JUnit logger instead of trying to convert the test results:
GitHub Actions do not support publishing test results! This is a pretty big shortcoming for a Continuous Integration system. A couple of third-party Actions are trying to fill the gap but if you’re multi-targeting or running your tests on multiple platforms expect additional complexity in you build script.
Visual Studio Test Results File (known colloquially as
TRX) gave me the best output but the Trx Logger has limited abilities in terms of naming test results files. By default the test results file names do not include the assembly or the framework version. Using multi-targeting and the
LogFileName parameter will result in test results files being overwritten by the last target. Using the recently introduced
LogFilePrefix parameter doesn’t solve the issue, token expansion is not supported so you’ll have to rename the files to include the assembly name and remove the trailing numbers.
Ultimately I had to invoke
dotnet test for each test project / framework combination. The end result is pretty neat:
Failed tests display the test runner output and the stack trace:
.NET is now leveraging the “new”
*.csproj system and this means:
- No more
- No more
- No more tears
The references (projects and packages) and the package configuration are both contained in the
*.csproj making it the single source of truth!
Referencing a project without turning it into a package reference
Sometimes you want to include a
DLL in a
NuGet package rather than add it as a package reference.
SuperLogic project depends on the
ExtraLogic project but we don’t want to ship
ExtraLogic as a package. Instead we want to include
Contoso.Hello.ExtraLogic.dll in the
SuperLogic package directly. Currently this is not supported out of the box but the team is tracking it.
Luckily this issue provides a workaround. All the modifications will take place in
- In the
<PropertyGroup>section add the following line:
- Prevent the project to be added as a package reference by making all assets private.
- Finally add the target responsible of copying the
The result is the following
And the assemblies have been versioned as expected:
Note: you can also use the “new”
*.csproj system for
NuGet packages targeting older
.NET Framework versions.
On any branches starting with
NuGet packages will be published to a pre-release feed. If the branch is
main it’ll be published to the production feed. This is handled by
GitHub Actions in this section of the configuration.
When publishing the packages, I’m also publishing the associated symbols to allow consumers to debug through my packages.
Create a GitHub release
By default the
GitHub CLI does not use the workflow token, you have to explicitly provide it using an environment variable:
GitHub CLI relies on the shell expanding wildcards which doesn’t happen on Windows. So instead of providing:
You’ll need to provide:
GitHub CLI doesn’t allow to create a release without release notes in a non-interactive scenario. I worked around this by creating an empty file and providing it to the
This is one possible workflow only. I’ve glossed over many details and taken some shortcuts (for example there is no support for Pull Request builds).
Those are the key takeaways:
- Do upfront planning on how you want to handle versioning. This is the hardest part and the one that will be the hardest to fix later on. Read the GitVersion documentation carefully before making any decision.
- Do what works for your team. If you didn’t have any issues with auto-incrementing your builds, keep doing so. There is no point bringing additional complexity to fix a problem you don’t have.
- Don’t assume you’ll be running on
Visual Studio Enterpriseinstalled. Adding cross-platform or other
Code…) support from the start will make your life easier down the track.