Continuous Integration in VSTS using .Net Core (with Code Coverage), NUnit, SonarQube: Part 3: VSTS SonarQube Build Task

What is SonarQube? From SonaQube’s WebsiteSonarQube provides the capability to not only show health of an application but also to highlight issues newly introduced. With a Quality Gate in place, you can fix the leak and therefore improve code quality systematically.”

In short, it’s a continuous integration process targeting developers to set triggers and/or thresholds on maintaining quality code using gates.

Here’s a high-level screenshot of what SonarQube has to offer (Actual screenshot of an application that went through SonarQube’s capabilities:

image

Note that the instance of SonarQube that I’ve used here is their SaaS based offering – SonarCloud. I didn’t want to go through the hassle of hosting my own instance of SonarQube rather use the SaaS based offering as a guideline. In my opinion, SaaS based offerings are better options for medium to enterprise size companies for multiple reasons (Cost, Support, Maintenance, etc…)

To see detailed description of what SonarQube has to offer: https://www.sonarqube.org/features/clean-code/

Personally, I love everything what SonarQube has to offer. Note that SonarQube can also be self-hosted, If you want to host SonarQube within your IT shop, you can step by step directions here: https://www.sonarqube.org/downloads/

Let’s go through setting up SonarQube in VSTS:

Step 1: Prepare analysis on SonarQube

NOTE: Make sure that this task comes before any application build task. This should be the first task. In my example, this task comes after restore Nuget step. This shouldn’t affect how the analysis works. Nuget restore is pretty much restoring Nuget packages for the given .Net solution/project(s).

This is the most crucial step of the process. This what sets all the properties in build time. The fields you need to enter here are both the Project Key and Project Name. These values can be obtained through SonarQube’s administration page or the landing page of your project in SonarQube.

One important field missing here is the Organization. This is needed to publish to SonarQube. As of writing this post, version 4.x of this task will fail unless you specifically add an additional property to set the organization. You set this by expanding “Advanced” on the task and typing:

sonar.organization=<Org Value>

Both Org and Project Keys are specified as well in the project landing page in SonarQube’s site.

image

Step 2: Run Code Analysis

This step should come after a successfully test task for your build. The results from the unit tests are gathered (including code coverage), analyzes the results and preps the proper files for publishing to SonarQube.

image

Step 3: Publish Quality Gate Result

This is the final step. It should come right after the Code Analysis task. No settings are done here since all settings have been properly set in the first step (Prepare analysis on SonarQube).

image

A successful build with SonarQube integration looks like this:

image

Continuous Integration in VSTS using .Net Core (with Code Coverage), NUnit, SonarQube: Part 2: VSTS Build Definition Setup – .Net Core and NUnit

If you haven’t setup your .Net Core project/s for code coverage instrumentation, see my previous post: Part 1: .Net Core Project Setup – Code Coverage.

That said, let’s go through the settings for enabling code coverage in VSTS builds. The basic structure of CI build definition would be:

  1. Build the application
  2. Run Tests (Unit Tests with Code Coverage)
  3. Publish Artifacts

In this post, I’ll skip over using NUnit as a test framework for .Net Core. For using NUnit as a test framework, see my previous post: Using NUNIT Test Framework do validate deployments in VSTS Release Management

image

Note from the image above that we’ve disabled the Dotnet test task because code coverage is currently not supported on dotnet.exe CLI (as of writing of this post)

However, vstest.console.exe does support code coverage. This is the task we’ve enabled to run our unit tests for code coverage instrumentation as well as running the tests from NUnit written code. Vstest.console.exe automatically detects NUnit tests since part of the restore nuget package includes the NUnit test adapters.

The important note here is to ensure that you properly setup the vstest task in the build definition and it’s settings:

  • Ensure that you have code coverage enabled option
  • Ensure that you are pointing to the .runsettings file for further code coverage settings
  • Install the NUnit adapters as part of your test project

Running the build yields the following result.

image

At this point, you can download the code coverage file and open the result in Visual Studio for further inspection.

You may have noticed that in my build, I also have tasks for SonarQube. What is SonarQube and why use it? For this, see part 3 (final) post for this series: Part 3: VSTS SonarQube Build Task

Continuous Integration in VSTS using .Net Core (with Code Coverage), NUnit, SonarQube: Part 1: .Net Core Project Setup – Code Coverage

There are 2 ways to discover and execute unit tests using Microsoft developed test harnesses:

  • Vstest.console.exe = This is the command-line used to execute tests within/embedded in Visual Studio IDE
  • Dotnet.exe = This is the command line interface (CLI) specific to .Net Core Projects

Documentation for Vstest.console.exe is documented here: https://msdn.microsoft.com/en-us/library/jj155796.aspx

For .Net Core Projects: https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test?tabs=netcore2x

The primary difference between both is that vstest.console.exe can execute tests developed in .Net Framework and .Net Core while dotnet.exe is specifically for .Net Core

An example of executing tests for the same assembly domain (test project) would be:

VSTest.Console.exe:

vstest.console.exe <testassembly>.dll (Pointer to the compiled Assembly)

Dotnet.exe:

dotnet test <testassemblyproject>.csproj (Pointer to the actual .Net Core Test Project)

The issue with dotnet.exe (CLI) is that Code Coverage doesn’t work. In order for code coverage to work on .Net Core projects, you need to:

  1. Edit the .Net Core projects you want to instrument for code coverage
  2. Use vstest.console.exe and supply /EnableCodeCoverage switch

Edit the .Net Core project/s for code coverage instrumentation

When you run unit tests in visual studio and select the option to “Analyze Code Coverage for Selected Tests” (as seen below), by default, code coverage results will not be captured.

image

As of writing of this post, the fix is to modify the project file and enable DebugType to Full on the propertygroup section of the project file.

image

Save the project file and run the unit tests again by selecting the option: to “Analyze Code Coverage for Selected Tests” and you’ll see similar results as shown below.

image

Use vstest.console.exe and supply /EnableCodeCoverage switch

As you saw within Visual Studio, running tests with code coverage can be trigged via a simple click on the context menu. If you want to execute your unit test with code coverage in a command line, you invoke /EnableCodeCoverage switch.

vstest.console.exe <testassembly>.dll /EnableCodeCoverage

The result would be an export of the code coverage results to a .coverage file. You can then open the file within Visual Studio to inspect the results. See screenshot below:

image

Setting up your .Net Core projects appropriately using the preceding steps should give you the proper code coverage numbers. More importantly, this allows you to seamlessly integrate with various build systems. Additionally, here are some tips and practices around code coverage:

Use a test .runsettings

Use a test .runsettings file to exclude assemblies you don’t want to instrument. The .runsettings file can be used on how tests are executed from vstest.console.exe. For more information, see the following: Configure unit tests by using a .runsettings file

Here an example on how you would want to exclude piece of code not to be measured for code coverage:

<DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Include>  
                <!-- Include all loaded .dll assemblies -->  
              </Include> 
              <Exclude>
                <!-- Exclude all loaded .dll assemblies with the words moq, essentially regex -->
                <ModulePath>.*\\[^\\]*moq[^\\]*\.dll</ModulePath>
                <ModulePath>.*\\[^\\]*Moq[^\\]*\.dll</ModulePath>
              </Exclude>
            </ModulePaths>
            <!-- We recommend you do not change the following values: -->
            <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <CollectAspDotNet>False</CollectAspDotNet>s
            <Attributes>
              <Exclude>
                <Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
              </Exclude>
            </Attributes>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

To use the .runsettings file, in Visual Studio, click on Test, Test Settings, Select Test Settings File (see below image)

SNAGHTML2d4683

[ExcludeFromCodeCoverage] attribute

Use [ExcludeFromCodeCoverage] attribute wherever appropriate. When a section of code is decorated with this attribute, that section of the code will be skipped for code coverage. Why? In certain cases, you don’t want code to be measured with code coverage. An example would be entity objects that have default property setters (get / set) that has no functionality. If there is “NO” logic developed on either the get and/or set property why measure it?

This ends the first part of this series, on the next part (VSTS Build Definition Setup – .Net Core and NUnit), we will hook up the test tasks in VSTS to include code coverage reporting.

Working With Stand Alone Entity Framework Core 2.0 in .Net Framework 4.6 (above) with SQL Server

When you work with EF Core, the initial project creation requires that you select a .Net Core project. This is all good if you’re entirely working with a .Net Core App. What about .Net Frameworks 4.6 and above? How about starting with a .Net Core app without EF Core?  There are multiple guides out there, scattered in multiple places to get EF Core installed and working. This post is intended to have you install EF Core at minimum and walk through the setup and unit testing scenarios.

As of this post, we’re utilizing Entity Framework 2.0 within .Net Framework 4.7 projects. Entity Framework Core 2.0 is compatible with versions of .Net Framework 4.6.x and above. Why use EF Core on .Net Framework projects? Simply put, compatibility reasons. Certain services in azure (Azure Functions for example) currently supports .Net Framework projects and not .Net Core. To circumvent the problem, the EF team has done a great job with EF Core so it can work with various .Net versions. While the intention of this post is adopting EF Core 2.0 within .Net 4.7, the goal is to show the features built in EF Core 2.0. In particular, I really love the capability to unit test databases through in memory channel. We’ll talk about this later.

If this is your first time working through Entity Framework, I strongly suggest going through the “Get Started Guide”, see the following article from Microsoft: https://docs.microsoft.com/en-us/ef/core/get-started/

Let’s get started: Create a .Net 4.7 Project

In Visual Studio, create a new project:

EF1

Install the following Nuget Packages for EF Core:

Microsoft.EntityFrameworkCore – Core EF libraries

Microsoft.EntityFrameworkCore.Design – The .NET Core CLI tools for EF Core

Microsoft.EntityFrameworkCore.SqlServer – EF Core Database Provider for SQL. In this case, we’ll be using SQL EF Core Provider

Microsoft.EntityFrameworkCore.Relational– EF Core Libraries that allows EF to be used to access many different databases. Some concepts are common to most databases, and are included in the primary EF Core components. Such concepts include expressing queries in LINQ, transactions, and tacking changes to objects once they are loaded from the database. NOTE: If you install Microsoft.EntityFrameworkCore.SqlServer, this will automatically install the Relational assemblies

Microsoft.EntityFrameworkCore.Tools – EF Core tools to create a model from the database

Microsoft.EntityFrameworkCore.SqlServer.Design– EF Core tools for SQL server

Microsoft.EntityFrameworkCore.InMemory – EF Core In-memory database provider for Entity Framework Core (to be used for testing purposes). This is or will be your best friend! One of the reasons why I switched to EF Core (besides it’s other cool features). You only need to install this package if you’re doing Unit Testing (which you should!)

Edit the Project Files:

Edit the project file and make sure the following entry appears in the initial property group.

<PropertyGroup> 
<AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
</PropertyGroup>

For test projects, also make sure the following entry is also present:

<GenerateBindingRedirectsOutputType>true</GenerateBindingRedirectsOutputType>

Implementing EF Core

I’m not going to go through the basics of EF Core. I’ll skip the entire overview and just dive deep on implementing EF entities and using the toolsets.

The Model and DBContext:
We’ll be using a simple model and context class here.

public class Employee
    {
        [Key]
        public int EmployeeId { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string DisplayName => FirstName + " " + LastName;
        public EmployeeType EmployeeType { get; set; }
    }

    public class EmployeeType
    {
        [Key]
        public int EmployeeTypeId { get; set; }
        public string EmployeeTypeRole { get; set; }
    }

public class DbContextEfCore : DbContext
    {
        public DbContextEfCore(DbContextOptions<DbContextEfCore> options) : base(options) { }

        public virtual DbSet<Employee> Employees { get; set; }
        public virtual DbSet<EmployeeType> EmployeeTypes { get; set; }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            //When creating an instance of the DbContext, you use this check to ensure that if you're not passing any Db Options to always use SQL. 
            if (!optionsBuilder.IsConfigured)
            {
                var connectingstring = ConfigurationManager.ConnectionStrings["SqlConnectionString"].ConnectionString;
                optionsBuilder.UseSqlServer(connectingstring);
            }
        }
    }

 

EF Tools

Database Migrations – This is far one of the best database upgrade tools that you can use for SQL server. Since we have our model and DBContext, let’s create the scripts to eventually create the database on any target server and update the database when necessary
Before you run any Database Migration commands, you’ll need to ensure that you have a class file that implements IDesignTimeDbContextFactory. This class will be recognized and used by entity framework to provide command line tooling such as code generation and database migrations. Before proceeding, make sure your application or web.config file has the connectingstring values set for your SQL server (see example below)

  <connectionStrings>
    <add name="SqlConnectionString" connectionString="Server=XXXX;Database=XXX;User ID=XXX;Password=XXX;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;" providerName="System.Data.SqlClient"/>
  </connectionStrings>

NOTE: Unfortunately, you cannot use InMemory Database when working with Database Migration Tools. In Memory Database doesn’t use a relational provider.

Here’s an example class file that you can use in your project:

public class DesignTimeDbContextFactory : IDesignTimeDbContextFactory<DbContextEfCore>
    {
        public DbContextEfCore CreateDbContext(string[] args)
        {
            var builder = new DbContextOptionsBuilder<DbContextEfCore>();
            //Database Migrations must use a relational provider. Microsoft.EntityFrameworkCore.InMemory is not a Relational provider and therefore cannot be use with Migrations.
            //builder.UseInMemoryDatabase("EmployeeDB");
            //You can point to use SQLExpress on your local machine
            var connectionstring = ConfigurationManager.ConnectionStrings["SqlConnectionString"].ConnectionString;
            builder.UseSqlServer(connectionstring);
            return new DbContextEfCore(builder.Options);
        }
    }

 

When running command line options, make sure you select the .Net Project where you are working on Entity Framework. At the same time, ensure that the default start-up project in solution explorer is set on the EF project

In Visual Studio. Go to Tools > Nuget Package Manager > Package Manager Console

In the PMC (Package Manager Console) type: Add-Migration InitialCreate

After running, this command, notice that it will create the proper class files for you to create the initial database schema.

EF2

Let’s create our database on the target server. For this, run: update-database. When done, you should see the following:

EF3

Your database has been created as well:

EF4

As we all know, requirements do change more often than before (specially in agile environments), a request was made to add the city and zip code to the employee data. This is easy as:

  • Modifying the entity (model)
  • Running “Add-Migration <ChangeSet>”
  • Running “Update-Database”

The Entity Change:

public class Employee
    {
        [Key]
        public int EmployeeId { get; set; }

        public string FirstName { get; set; }

        public string LastName { get; set; }

        public string DisplayName => FirstName + " " + LastName;

        public string City { get; set; }

        public int ZipCode { get; set; }

        public EmployeeType EmployeeType { get; set; }
    }

In the PMC (Package Manager Console) type: Add-Migration AddCityAndZipCodeToEmployee

New file has been created:

EF5

Schema has been added as well:

EF6

In the PMC (Package Manager Console) type: Update-Database

EF7

Changes have been applied to the database directly.

EF8

I’ll leave database migrations here. You can get more information on all the nice and nifty features of database migrations here: https://msdn.microsoft.com/en-us/library/jj554735(v=vs.113).aspx

For all command line options for database migrations: https://docs.microsoft.com/en-us/ef/core/miscellaneous/cli/powershell

Finally, Database Migrations uses the __MigrationsHistory table to store what migrations have been applied to the database.

Testing EF Core

This is my favorite topic! I can’t stress enough how easy it is to Unit Test databases using EF Core. InMemory Database as part of EF Core sets up the runtime and everything else you need to Unit Test databases. Before EF Core, it really took sometime to setup mock objects, fakes, dependencies, etc… With EF Core InMemory Database, I was able to focus more on the design of the database rather spend time focusing on the test harness. The short story is this, InMemory database is another provider in EF that stores data In Memory during runtime. Meaning, you get all the same benefits, features and functions with EF to SQL (or other provider) except with the beauty of not connecting to an actual provider endpoint (SQL in this case).

Start of by adding a new .Net Framework 4.7 Test Project

EF9

Add a reference to the previously created project with EF Core Enabled.

Add the following nuget packages:

  • Microsoft.EntityFrameworkCore
  • Microsoft.EntityFrameworkCore.InMemory
  • Microsoft.EntityFrameworkCore.SqlServer

Edit the project file and make sure the following entry appears in the initial property group.

<PropertyGroup>
<AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
<GenerateBindingRedirectsOutputType>true</GenerateBindingRedirectsOutputType>
</PropertyGroup>

Test Class:

[TestClass]
    public class DbContextTests
    {
        protected DbContextOptions<DbContextEfCore> Dboptionscontext;

        [TestInitialize]
        public void TestInitialize()
        {
            //This is where you use InMemory Provider for working with Unit Tests
            //Unlike before, we're you'll need to work with Mocks such as MOQ or RhinoMocks, you can use InMemory Provider
            Dboptionscontext = new DbContextOptionsBuilder<DbContextEfCore>().UseInMemoryDatabase("EmployeeDatabaseInMemoery").Options;

            //Lets switch to use Sql Server
            //var connectionstring = ConfigurationManager.ConnectionStrings["SqlConnectionString"].ConnectionString;
            //Dboptionscontext = new DbContextOptionsBuilder<DbContextEfCore>().UseSqlServer(connectionstring).Options;
        }

        [TestMethod]
        public void ValidateInsertNewEmployeeRecord()
        {
            //Employee Entity Setup
            var employee = new Employee
            {
                FirstName = "Don",
                LastName = "Tan",
                City = "Seattle",
                ZipCode = 98023,
                EmployeeType = new EmployeeType
                {
                    EmployeeTypeRole = "HR"

                }
            };

            using (var dbcontext = new DbContextEfCore(Dboptionscontext))
            {
                dbcontext.Employees.Add(employee);
                dbcontext.SaveChanges();
                Assert.IsTrue(dbcontext.Employees.Any());
            }
        }
    }

Test Initialize method is where I set the DBContext option to use InMemory vs SQL providers. When you run the tests switching either InMemory or SQL DBContext options, both tests run successfully.

Switching to Sql EF Core provider, stores the data to the DB directly:

EF10

Using InMemory provider for EF Core lets you focus unit testing your database more efficiently. This is definitely useful when working with multiple tables and each table has multiple relationships (keys and other constraints). More importantly, this lets you focus designing the appropriate DB strategies such as repository or unit of work patterns.

Using NUNIT Test Framework do validate deployments in VSTS Release Management

As of writing of this post, I prefer NUnit when using .Net for developing application deployment tests. Why? Simply put, better support and test execution. More importantly, the ability to simply reference test runners without installing the actual test execution files (Like VSTest or MSTest). This guide will walk you through on developing simple tests via NUnit, build the artifacts then executing those tests on the server where the application is being deployed to.

NOTE: NUnit and MSTest are completely separate testing frameworks. You can mix/reference both NUnit and MSTest Test frameworks in the same VS Test Project and Tests (Test Methods) without duplicating the same tests. This however is a separate post.

Backing a bit, we use VSTS as our primary adoption platform for Continuous Integration and Deployment. In short, CI through build definitions and CD through release management. We have standards within our processes and one of those standards is to ensure that any application being deployed has the appropriate application deployment tests. Nope, I’m not talking about smoke tests where an application is being verified “after” its been deployed through a UI tests or simply some high level scenarios. I refer to “application deployments tests” as tests developed to be run specifically on the server where the application is being deployed to. This mostly maps to any application being deployed and hosted to individual servers (IIS hosted machines if you’re a Windows shop)

Let’s begin.

1) Develop NUnit Tests to validate application deployment.

Before proceeding, you need to ensure that you at least have one application deployment tests. The number of tests vary on the application that you want to verify. In my scenario, I’m deploying a Web API so one of the tests to validate is whether a web method works after deployment.

Here, we’re going to start off with an existing Visual Studio Test Project. However, If this is your first time (as a devops person), feel free to go through this guide, “Create a unit test project

Reference NUnit assemblies in your test project.

Expand your Test Project, right click on “References” and click on “Manage NuGet Packages”. The nuget package explorer window opens and type in “Nunit” under the search box within the “Browse” tab.

Nunit1

NOTE: If you want to see your NUnit Tests within Visual Studio, you’ll need to install the NUnit Test Adapter within Visual Studio, “Tools”, “Extensions and Updates

Decorate your code with the NUnit Test Attributes

Nunit2

The goal for running tests is to ensure that the app has been deployed properly. In these situations, tests would then normally target or construct the URL based on the IP Address or the HostName of the sever then the application URL. This would be something that you’ll need to plan on how to build your application URLs correctly. There are many ways to do this and I’ll creating a separate blog post for this.

Once you’ve validated your tests targeting the right server, we then package the tests as artifacts to be executed on the server. For this, see the next step.

2) Package your tests files as build artifacts to be consumed in VSTS Release Management

In normal CI processes, your tests files should be coupled together with your application deployment files. In an event that your tests files are not packages together with your application deployments files, follow these steps:

Start off by creating a build definition in VSTS. For this, follow this guide. “Build and deploy your app”.

Build your Test Solution with the Test Project

Nunit3

Publish your Test Artifacts/Files

Nunit4

NOTE: Very Important! Make sure your Test Files includes the NUnit Console Runner assemblies and executable. There are many ways to reference the Nuget Test Runner files but for the purpose of this demo, ensure that your Test artifacts include all test runner files. Here’s a snipped of such files.

Nunit5

3) Modify your release tasks to include running application deployment tests via NUnit

In your existing release definition:

Add the build artifacts (test artifacts) as part of your Release Definition

Nunit7

Add a Command Line Task to Run NUnit Tests

Nunit6

And the result…

Nunit8

Referencing MSTest And MSTestv2 Unit Testing Framework Through Namespace Aliasing

Let me start-off by explaining what MSTest and MSTestV2 are.

MSTest (Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll) – This is the unit testing framework that comes pre-installed when you install Visual Studio IDE (Available through the .Net Framework – GAC)

MStestV2 (Microsoft.VisualStudio.TestPlatform.TestFramework.dll) – This is now the open source version of MSTest. With any open source libraries, there are lots of good contributions but also features do change more frequent. More often, removed (or enhanced in this case). You install this version of MSTest through Nuget.

With that brief description on MSTest and MSTestV2, now comes the question: Why would I reference both MSTest and MSTestV2 in the same test project? Well, there are two reasons; Backwards compatibility and issues exposed in MSTestV2 that is still being worked on.

In terms of backwards compatibility, I work with many developers around utilizing data driven features in MSTest. The good back then is that we can data drive tests using many data source providers (e.g. Excel, SQL, etc…). The bad part is that the open source framework (MSTestV2) only supports both XML and CSV as the data source providers (Though, it supports DataRow as a data source which is good).

Ideally, I would ask the developers to migrate directly to MSTestV2 but in this case, I’d like for them to regress any issues they find in MSTest and see what else could break in MSTestV2.

The issue: Referencing both dlls causes collisions and/or conflicts simply because most of the attributes (or All- [TestClass],[TestMethod],etc…) uses the exact same namespace:

Microsoft.VisualStudio.TestTools.UnitTesting

The Solution! Welcome back namespace aliasing. The last time I used namespace aliasing, oh, I can’t remember exactly but probably late 2006 (C# 2.0)

With namespace aliases, you can reference multiple assemblies even if those assemblies have the exact same namespace

Step 1: Provide an alias name at the assembly level.

Go to the properties of each assembly and provide an alias.

MSTest1

Step 2: In code, refer to the assembly alias using the C# reserved keywordextern

extern alias FrameworkV1;
extern alias FrameworkV2;

using System;
using TestFrameworkV1 = FrameworkV1.Microsoft.VisualStudio.TestTools.UnitTesting;
using TestFrameworkV2 = FrameworkV2.Microsoft.VisualStudio.TestTools.UnitTesting;

Step 3: Refer to the appropriate assembly classes and/or attributes through the namespace alias (variable you created through the “using” statement)

MSTest2

And the result

MSTest3

Unit Tests (TDD) + Code Coverage = “Happy Couple”

We all rave and talk about TDD (Test Driven Development) all the time. Have you asked yourself these questions?

· “Do my unit tests truly cover blocks (or lines) of code that I’ve implemented?”

· “How do I ensure that specific features (implementation) is doing what it’s supposed to do?”

· “Is there a possibility that a block or line of code that I’ve written is not being touched by my unit tests?”

This post however, I’m not going through the practices and understanding of how unit testing works. There are many resources and literature available for you to look at (just google TDD J). Most of you already know how to do this but I do want to share my experience and practices around unit testing “WITH” code coverage. Have you used code coverage before? If not, let’s start with that.

So, what is code coverage? Simply put: “it is a measure (%) used to describe the degree to which the source code of a program is executed when a particular test suite runs”

Source: https://en.wikipedia.org/wiki/Code_coverage

We also describe that a program/application with high degree of code coverage, has a lower chance of containing undetected software defects compared to a program/application with low code coverage, again depending on the test suite. It’s easy to produce tons of tests that should cover the code, but we normally measure it. Covering code just means you need quality tests that verifies the functionality of blocked or line code that you wrote (the quantity isn’t that important). Which boils down to, you are not writing “Regression Tests” (validating edge cases and/or test case families) when writing unit tests to measure code coverage. However, you may have requirements to implement certain rules that may touch edge case scenarios. In this case, you write unit tests for those because it’s now a logic/function that you will implement.

Finding the sweet spot! When is “Enough” enough? It’s when you can make changes to your code with confidence that you’re not breaking anything and to me what it means is that you have tested a block of code that has logic and/or implementation in place. As a .Net developer, I come to wonder, how about properties? Specially auto generated properties? Do we account for code coverage numbers for those? My answer to this is No. Auto generated properties or properties in general and by default doesn’t have logic in place. Hence, why write unit tests for something that doesn’t have logic?, Then why measure it through code coverage?

Here, I’ll start off with a project that has a unit test available and some level of functionality. Consider the following unit test.

Consider the following unit test code block:

[TestMethod]
        public void ValidateGetRequest()
        {
            var uri = new Uri("https://api.github.com/users/mikelo/repos");
            var jsonresponse = new HttpConnectionService().GetResponse(uri);
            Assert.IsTrue(jsonresponse.Contains("38358544"));
        }

This unit test validates getting a JSon response from a web api. In this case, a valid public API from Github. For simplicity, I simply want to do an HTTP GET from this web api to get repo’s (Git repositories) from a Github contributor.

Below is the implementation

public string GetResponse(Uri url)
        {
            string jsonresponse;
            using (var client = new HttpClient())
            {
                client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
                client.DefaultRequestHeaders.Add("User-Agent", "client agent");
                var response = client.GetAsync(url).Result;
                jsonresponse = response.Content.ReadAsStringAsync().Result;

            }
            return jsonresponse;
        }

Running the tests yields the following results:

TDDCC1

So far so good, we’ve achieved writing a unit test to validate a response from a public web api as well as ensured that the test we wrote covered 100% of the code implementation.

A problem arises from this. It’s apparent that we have a major dependency in our unit test. Now and days we use and rely on build systems (such as Jenkins and VSTS) to compile, tests and publish artifacts. The practice behind Unit Testing is to ensure that all unit tests executed are de-coupled and not have any dependencies in place.

Besides dependencies, I’m sure by now you’ve realized that there’s no guarantee that every single call you make will be successful so now comes the next cycle of our work, let’s expand the implementation to include a try catch block so at some point we can customize what we want to return or throw back to the user in an event a problem occurs

We start by creating a new test to validate that any exceptions thrown are caught in the catch block. Again, TDD circle of:

TDDCC2

We wrote the Unit Test, failed it and now need to pass it. How do we pass a unit test where we expect an exception and still make it pass? In MSTESTV2, we can apply a method attribute to a test method that expects a type of exception. Once used, this tells the test method that the test “SHOULD” pass when an exception is caught of an exception type. Here’s the newly created unit test:

        [TestMethod]
        [ExpectedException(typeof(HttpRequestException))]
        public void ValidateGetRequestCatchesException()
        {
            var uri = new Uri("https://apiXXXX.github.com/users/mikelo/repos");
            var jsonresponse = new HttpConnectionService().GetResponse(uri);
        }

Notice that the URL was changed to point to a non-existent URL. We also refactored implementation on the code by adding a try – catch block and with this line: response.EnsureSuccessStatusCode();

public string GetResponse(Uri url)
        {
            try
            {
                string jsonresponse;
                using (var client = new HttpClient())
                {
                    client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
                    client.DefaultRequestHeaders.Add("User-Agent", "client agent");
                    var response = client.GetAsync(url).Result;
                    response.EnsureSuccessStatusCode();
                    jsonresponse = response.Content.ReadAsStringAsync().Result;
                }
                return jsonresponse;
            }
            catch (Exception e)
            {
                Debug.WriteLine(e);
                throw;
            }
        }

Using this method: EnsureSuccessStatusCode() ensures that exceptions are thrown if the IsSuccessStatusCode property for the HTTP response is false. This is my own implementation and I’m sure there are many ways to catch and throw errors/exceptions back to users.

So far so good, I have 2 Unit Tests that:

· Passes

· Verifies implementation of the code I wrote is being covered through Code Coverage – 100%

(Note: The blue highlighted section of the screenshot)

TDDCC3

While we’ve satisfied basic principles of TDD + Code Coverage thus far, we’re still left with the point of isolation. Current TDD practice suggests that any unit test when executed should be isolated and no real dependencies should be called upon.

Our Unit Tests still rely on a working API endpoint. This is problematic as we know unit tests are autonomous.

Solution? Mocking/Faking endpoints! The best part I love about TDD is that inevitably your code leads to better design through Dependency Injection and/or Inversion of Control.

Before I go further, it might be worthwhile to talk about Mocking (Faking) and Dependency Injection. I’ll briefly show you how dependency injection works later.

What is Mocking? Mocking is primarily used in unit testing. It is used to isolate behavior of an object or function of what you want to test by simulating the behavior of the actual object and/or function.

There are so many articles, guides and practices around mocking. There are 2 well known mocking frameworks that are used by many developers

1) MOQ – My favorite. Easy to use, open source with many contributors. MOQ is hosted in GitHub and works well in .Net. https://github.com/Moq/moq4/wiki/Quickstart

2) RhinoMocks – Same concept as MOQ. https://hibernatingrhinos.com/oss/rhino-mocks

What is Dependency Injection? If you’ve been practicing TDD, I’m quite certain your code will eventually lead to dependency injection.

Snippet from Wikipedia:

https://en.wikipedia.org/wiki/Dependency_injection

“Dependency injection is a technique whereby one object supplies the dependencies of another object. A dependency is an object that can be used (a service). An injection is the passing of a dependency to a dependent object (a client) that would use it. The service is made part of the client’s state, passing the service to the client, rather than allowing a client to build or find the service, is the fundamental requirement of the pattern.”

For .Net developers: Dependency Injection implements Interfaces then used in class constructors as parameters.

Let’s go back to our code and will consider dependency injection later.

Unit Tests –CHECK. Code Coverage – CHECK. As I write this post, I’m in route to San Francisco. What a great way to work on TDD and ensure I start mocking at this point. Given that I don’t have any persistent connection to the public API my tests will fail

TDDCC4

Luckily, the implementation I decided to use for API request is HTTPClient (built in .Net) which allows me to pass in an “HttpMessageHandler” which therefore I can mock the data.

With a little bit of refactor work: (Again, Red -> Green -> Refactor analogy), Here’s the modified version of the HttpConnectionService class

public string GetResponse(Uri url, HttpMessageHandler handler = null)
        {
            try
            {
                string jsonresponse;
                using (var client = handler == null ? new HttpClient() : new HttpClient(handler))
                {
                    client.Timeout = TimeSpan.FromSeconds(3);
                    client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
                    client.DefaultRequestHeaders.Add("User-Agent", "client agent");
                    var requestmessage = new HttpRequestMessage
                    {
                        Method = HttpMethod.Get,
                        RequestUri = url
                    };
                    var response = client.SendAsync(requestmessage).Result;
                    response.EnsureSuccessStatusCode();
                    jsonresponse = response.Content.ReadAsStringAsync().Result;
                }
                return jsonresponse;
            }
            catch (Exception e)
            {
                Debug.WriteLine(e);
                throw;
            }
        }

Here’s a modified version of the Unit Tests:

[TestMethod]
        public void ValidateGetRequest()
        {
            //No Need to specify a valid URI. I'm mocking the "state" or behavior at this point.
            //var uri = new Uri("https://api.github.com/users/mikelo/repos");
            var mockhandler = new Mock<HttpMessageHandler>();
            mockhandler.Protected()
                .Setup<Task<HttpResponseMessage>>("SendAsync", ItExpr.IsAny<HttpRequestMessage>(), ItExpr.IsAny<CancellationToken>())
                .Returns(Task<HttpResponseMessage>.Factory.StartNew(() => new HttpResponseMessage
                {
                    StatusCode = HttpStatusCode.OK,
                    Content = new StringContent("Mock Response. OK", Encoding.UTF8, "application/json")
                }));
            var jsonresponse = new HttpConnectionService().GetResponse(new Uri("http://someuri"), mockhandler.Object);
            Assert.IsTrue(jsonresponse.Contains("Mock Response. OK"));

Passing the first test yields the following code coverage result:

TDDCC5

I passed the first unit test because I was able mock the request however this test doesn’t cover all the code blocks. Code Coverage is at 89.6%. Great progress so far! Note that code coverage % is subjective to the total code blocks developed. Importantly in this picture, the specific Unit Test didn’t go through the exception block. Let’s fix the other tests to validate exceptions are caught. Below is the modified version of the Unit Test to validate exception handling

[TestMethod]
        [ExpectedException(typeof(HttpRequestException))]
        public void ValidateGetRequestCatchesException()
        {
            var mockhandler = new Mock<HttpMessageHandler>();
            mockhandler.Protected()
                .Setup<Task<HttpResponseMessage>>("SendAsync", ItExpr.IsAny<HttpRequestMessage>(),
                    ItExpr.IsAny<CancellationToken>())
                .Throws<HttpRequestException>();
            var jsonresponse = new HttpConnectionService().GetResponse(new Uri("http://someuri"), mockhandler.Object);
        }

Both test passes! Code Coverage went up to: 96.55%! Lastly, both tests reached relevant code blocks in implementation.

TDDCC6

Why wasn’t I able to achieve 100% code coverage even though the Unit Tests covered the path to all code blocks? It seems that if use certain built .net features (highlighted in yellow) it treats it as an ambiguous state. I would only assume at this point that it’s how code coverage works in Visual Studio and/or the libraries for Code Coverage. In this case, MSTest.

At this point, it’s debatable that 100% code coverage should be met all the time. In the last refactor work, we didn’t meet 100% code coverage but it’s acceptable in this case. I would probably bet that as you continue to practice TDD (now with code coverage in mind J), you will have an acceptable range of code coverage %. Meaning, not all the time you will attain 100% code coverage

The real point here is that we’ve met the criteria for TDD by ensuring unit tests written validates any change or refactor work. This is what TDD + Code Coverage does. With Code Coverage, you ensure that any change or refactor work you do, has valid unit tests which is shown through code coverage numbers

 

A modified version of TDD circle would show:

TDDCC7

 

 

Now let’s expand the code a bit by introducing DI (Dependency Injection) I’ll add an abstraction layer (Façade) so that the user doesn’t call the connection service directly rather call this façade class for working with data. This is a common practice so the façade layer can work with any other business logic or requirement. Here’s the new class implementation:

public class GithubApiService
    {
        private readonly IConnectionService _connectionService;
        public GithubApiService(IConnectionService connectionService)
        {
            _connectionService = connectionService;
        }

        public string GetReposFromGitHub(Uri uri, HttpMessageHandler handler = null)
        {
            var response = _connectionService.GetResponse(uri, handler);
            return $"From GitHubApiService Class. Response Value: {response}";
        }
    }

I used DI to pass the IConnectionService in the constructor. When I do this:

1) I tell the object (constructor parameter) that during instantiation, I pass in an interface of IConnectionService type.

2) I can then mock the data directly in the façade layer instead of the ConnectionService layer.

As you might guess, here’s the Unit Test for validating the new façade class.

[TestMethod]
        public void ValidateGithubApiService_GetReposFromGitHub()
        {
            var mockhandler = new Mock<IConnectionService>();
            mockhandler.Setup(service => service.GetResponse(It.IsAny<Uri>(),null)).Returns("Mock Response. OK");
            var githubapiservice = new GithubApiService(mockhandler.Object);
            var response = githubapiservice.GetReposFromGitHub(new Uri("http://someuri"));
            Assert.IsTrue(response.Contains("From GitHubApiService Class. Response Value: Mock Response. OK"));
        }

Final Screenshot!

TDDCC8

Here’s an actual usage of the façade layer without mocking data:

[TestMethod]
        public void ValidateGithubApiService_GetReposFromGitHubWIthConnectionService()
        {

            var githubapiservice = new GithubApiService(new HttpConnectionService());
            var response = githubapiservice.GetReposFromGitHub(new Uri("https://api.github.com/users/mikelo/repos"));
            Assert.IsTrue(response.Contains("38358544"));
        }

There’s a lot more information on Dependency Injection. This post is not intended to deep dive on DI or Mocking rather ensure that when we practice TDD, we should also account for Code Coverage as measure of quality.