Continuous Integration in VSTS using .Net Core (with Code Coverage), NUnit, SonarQube: Part 1: .Net Core Project Setup – Code Coverage

There are 2 ways to discover and execute unit tests using Microsoft developed test harnesses:

  • Vstest.console.exe = This is the command-line used to execute tests within/embedded in Visual Studio IDE
  • Dotnet.exe = This is the command line interface (CLI) specific to .Net Core Projects

Documentation for Vstest.console.exe is documented here:

For .Net Core Projects:

The primary difference between both is that vstest.console.exe can execute tests developed in .Net Framework and .Net Core while dotnet.exe is specifically for .Net Core

An example of executing tests for the same assembly domain (test project) would be:


vstest.console.exe <testassembly>.dll (Pointer to the compiled Assembly)


dotnet test <testassemblyproject>.csproj (Pointer to the actual .Net Core Test Project)

The issue with dotnet.exe (CLI) is that Code Coverage doesn’t work. In order for code coverage to work on .Net Core projects, you need to:

  1. Edit the .Net Core projects you want to instrument for code coverage
  2. Use vstest.console.exe and supply /EnableCodeCoverage switch

Edit the .Net Core project/s for code coverage instrumentation

When you run unit tests in visual studio and select the option to “Analyze Code Coverage for Selected Tests” (as seen below), by default, code coverage results will not be captured.


As of writing of this post, the fix is to modify the project file and enable DebugType to Full on the propertygroup section of the project file.


Save the project file and run the unit tests again by selecting the option: to “Analyze Code Coverage for Selected Tests” and you’ll see similar results as shown below.


Use vstest.console.exe and supply /EnableCodeCoverage switch

As you saw within Visual Studio, running tests with code coverage can be trigged via a simple click on the context menu. If you want to execute your unit test with code coverage in a command line, you invoke /EnableCodeCoverage switch.

vstest.console.exe <testassembly>.dll /EnableCodeCoverage

The result would be an export of the code coverage results to a .coverage file. You can then open the file within Visual Studio to inspect the results. See screenshot below:


Setting up your .Net Core projects appropriately using the preceding steps should give you the proper code coverage numbers. More importantly, this allows you to seamlessly integrate with various build systems. Additionally, here are some tips and practices around code coverage:

Use a test .runsettings

Use a test .runsettings file to exclude assemblies you don’t want to instrument. The .runsettings file can be used on how tests are executed from vstest.console.exe. For more information, see the following: Configure unit tests by using a .runsettings file

Here an example on how you would want to exclude piece of code not to be measured for code coverage:

      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
                <!-- Include all loaded .dll assemblies -->  
                <!-- Exclude all loaded .dll assemblies with the words moq, essentially regex -->
            <!-- We recommend you do not change the following values: -->

To use the .runsettings file, in Visual Studio, click on Test, Test Settings, Select Test Settings File (see below image)


[ExcludeFromCodeCoverage] attribute

Use [ExcludeFromCodeCoverage] attribute wherever appropriate. When a section of code is decorated with this attribute, that section of the code will be skipped for code coverage. Why? In certain cases, you don’t want code to be measured with code coverage. An example would be entity objects that have default property setters (get / set) that has no functionality. If there is “NO” logic developed on either the get and/or set property why measure it?

This ends the first part of this series, on the next part (VSTS Build Definition Setup – .Net Core and NUnit), we will hook up the test tasks in VSTS to include code coverage reporting.


Referencing MSTest And MSTestv2 Unit Testing Framework Through Namespace Aliasing

Let me start-off by explaining what MSTest and MSTestV2 are.

MSTest (Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll) – This is the unit testing framework that comes pre-installed when you install Visual Studio IDE (Available through the .Net Framework – GAC)

MStestV2 (Microsoft.VisualStudio.TestPlatform.TestFramework.dll) – This is now the open source version of MSTest. With any open source libraries, there are lots of good contributions but also features do change more frequent. More often, removed (or enhanced in this case). You install this version of MSTest through Nuget.

With that brief description on MSTest and MSTestV2, now comes the question: Why would I reference both MSTest and MSTestV2 in the same test project? Well, there are two reasons; Backwards compatibility and issues exposed in MSTestV2 that is still being worked on.

In terms of backwards compatibility, I work with many developers around utilizing data driven features in MSTest. The good back then is that we can data drive tests using many data source providers (e.g. Excel, SQL, etc…). The bad part is that the open source framework (MSTestV2) only supports both XML and CSV as the data source providers (Though, it supports DataRow as a data source which is good).

Ideally, I would ask the developers to migrate directly to MSTestV2 but in this case, I’d like for them to regress any issues they find in MSTest and see what else could break in MSTestV2.

The issue: Referencing both dlls causes collisions and/or conflicts simply because most of the attributes (or All- [TestClass],[TestMethod],etc…) uses the exact same namespace:


The Solution! Welcome back namespace aliasing. The last time I used namespace aliasing, oh, I can’t remember exactly but probably late 2006 (C# 2.0)

With namespace aliases, you can reference multiple assemblies even if those assemblies have the exact same namespace

Step 1: Provide an alias name at the assembly level.

Go to the properties of each assembly and provide an alias.


Step 2: In code, refer to the assembly alias using the C# reserved keywordextern

extern alias FrameworkV1;
extern alias FrameworkV2;

using System;
using TestFrameworkV1 = FrameworkV1.Microsoft.VisualStudio.TestTools.UnitTesting;
using TestFrameworkV2 = FrameworkV2.Microsoft.VisualStudio.TestTools.UnitTesting;

Step 3: Refer to the appropriate assembly classes and/or attributes through the namespace alias (variable you created through the “using” statement)


And the result


Working with Entity Framework 6.0 ON IBM Informix V11.10+ in Visual Studio 2015

We know EF (Entify Framework) has many benefits working with Databases. Particularly from a development and performance stand-point. There are 2 versions of client connectivity SDK’s for working with Informix Databases:

IBM.Data.Informix.dll— Also referred to as the Common IDS .NET Provider. This assembly has been specifically created to help existing applications that were developed using the CSDK .NET Provider (SQLI protocol) to use the latest DRDA protocol support. It has additional support for some of the earlier Informix client features and is targeted only for .NET application development for Informix.

IBM.Data.DB2.dll— Also referred to as the DB2 .NET Provider. Although the name of the provider indicates DB2, it is in fact the single .NET provider for IBM database servers including DB2 and Informix. It is the recommended and preferred .NET provider for all clients targeting DB2 and new application development targeting Informix (Version 11.10 or later).

These are referenced from IBM’s website:

IBM.Data.DB2.dll is the preferred approach and uses Entity Framework. More importantly, this is the version that IBM will support for new enhancements in conjunction with Entity Framework.

Note: In order to use the .Net Data Provider (DB2) for Entity Framework 6.0, you need to ensure that DRDA protocol has been enabled on the Informix Server. For more information on DRDA overview and troubleshooting, see the following articles:

Overview of DRDA

TCPIP communication errors with DRDA

To get started with the .Net Data Provider for IBM Informix V11.10+ in Visual Studio 2015

1) Download and install the latest updates for Visual Studio 2015 (As of writing this blog post, the current updates is version 3)

2) Download and Install the DSDriver Package (Data Server Driver Package) from IBM’s site:

Specifically, this version” Special Build 35279 for IBM Data Server Driver Package (Windows/x86-64 64 bit) V10.5 Fix Pack 6” (special_35279_ntx64_dsdriver_EN.exe)

NOTE: As of writing on this blog, there could be more fix pack versions of the DS Driver Package, however the fix pack version above works well in VS 2015

3) Download and Install VSAI (IBM Database Add-Ins for Visual Studio) from IBM’s site:

Specifically, this version” Special Build 35192 for IBM Database Add-Ins for Visual Studio (Windows/x86-32 32 bit) V10.5 Fix Pack 6” (

NOTE: As of writing on this blog post, none of the Add-Ins for Visual Studio work on machine running Windows 10. IBM hasn’t provided a solution for this problem. Also, do not use DS Driver Package V11. Use DS Driver Package version 10.5+. This version is specifically compiled for EF 6.0

4) Install IBM Entity Framework 6.0 in your projects. Right click on the Project and Select “Manage Nuget Packages” and install latest EntityFramework.IBM.DB2



Sample Project to verify that you can use EF 6.0 connecting to IBM Informix Database (V11.10+):

  • Start of by creating a sample project in Visual Studio 2015. Any project would be fine, however for testing purposes, create a test project. This will allow you to generate Unit Tests to verify EF on Informix Database V11.10+
  • Right click on the project and Select “Add” then “New Item”.
  • From the list of items, select “ADO.NET Entity Data Model”, “Select IBM DB2 and IDS Servers



  • Follow the wizard but on the first step, select: “EF Designer from database
  • Click on “New Connection” and provide the proper server settings for the Informix DB server. Note that DRDA protocol needs to be enabled on the target server. Refer back to the top section of this post.


  • Click “OK” then click on “Next
  • Select the appropriate Tables then click “Finish

Once completed, your test project should have generated EF files which you can use to connect and work with Informix Server. Here’s an example of an auto generated file which is the actual context file that “inherits” DbContext from EF


The tools in VS also generates the entities for you. Since we’ve used the “EF Designer from database” template, all the of table to entity mappings is actually stored in the .EDMX file. You can explore this file visually or to see raw data, open the file in a text editor such as notepad


As an example, I’ve written some unit tests to verify some data from a table in Informix V11.10+.


With EF, we can greatly improve how are applications integrate on Informix. We can have better design principles and patterns using EF. A common design pattern with Databases is called “UNIT of Work”. Here’s a great article on how to implement Unit of Work design pattern with ASP.Net MVC

MSTEST: Extending Data Driven Tests to use IENUMERABLE<Object> as the Data Source

One great feature that I like in NUnit is the capability to use collection types for data driven tests. Meaning, you don’t have to open up an external data source connection, pull data and use it to drive parameters for your tests. With a simple attribute in NUnit, you can drive tests as indicated here:


NUnit implementation allows you to enumerate from a collection to data drive your tests. MSTest has the same extensibility and is outlined in the following blog:

Extending the Visual Studio Unit Test Type

From this blog, I was able to go through the steps and process how MSTest invokes and passes objects in a test method. When MSTest executes, the flow goes through:

  1. TestClassExtensionAttribute calls : GetExecution()
  2. TestExtensionExecution calls : CreateTestMethodInvoker(TestMethodInvokerContext context)
  3. ITestMethodInvoker calls : Invoke(params object[] parameters)


Throughout this process, you can use custom attributes and utilize attribute properties for passing in test data. Its best that you use custom attributes in the ITestMethodInvoker.Invoke()

In my solution, I want to develop a fast way of invoking IEnumerable<object> as my test data. In this case, I’ll consume custom attributes to provide a classname and methodname to return test data through reflection. I’ll then use that in ITestMethodInvoker.Invoke() to enumerate objects for my tests.

  • ClassName: class holding the method to generate test data
  • DataSourceName: method within the class that generates any test data

Project Setup:

Make sure that you have the following references in your project:

  • Microsoft.VisualStudio.QualityTools.Common.dll
  • Microsoft.VisualStudio.QualityTools.UnitTestFramework
  • Microsoft.VisualStudio.QualityTools.Vsip.dll

These assemblies are included as part of the .Net framework. Simply browse in the references section in your project

The Custom Attribute:

    [global::System.AttributeUsage(AttributeTargets.Method, Inherited = false, AllowMultiple = false)]
    public class EnumurableDataSourceAttribute : Attribute
        public string DataSourceName { get; set; }
        public string ClassName { get; set; }

        public EnumurableDataSourceAttribute(string className, string dataSourceName)
            this.DataSourceName = dataSourceName;
            this.ClassName = className;

Test Class Implementation:

    public class TestClassCollectionAttribute : TestClassExtensionAttribute
        public override Uri ExtensionId => new Uri("urn:TestClassAttribute");

        public override object GetClientSide()
            return base.GetClientSide();

        public override TestExtensionExecution GetExecution()
            return new TestExtension();

Test Extension:

public class TestExtension : TestExtensionExecution
        public override void Initialize(TestExecution execution)

        public override ITestMethodInvoker CreateTestMethodInvoker(TestMethodInvokerContext context)
            return new TestInvokerMethodCollection(context);

        public override void Dispose()

Test Method Invoker:

public class TestInvokerMethodCollection : ITestMethodInvoker
        private readonly TestMethodInvokerContext _context;
        public TestInvokerMethodCollection(TestMethodInvokerContext context)
            Debug.Assert(context != null);
            _context = context;
        public TestMethodInvokerResult Invoke(params object[] parameters)
            Trace.WriteLine($"Begin Invoke:Test Method Name: {_context.TestMethodInfo.Name}");
            Assembly testMethodAssembly = _context.TestMethodInfo.DeclaringType.Assembly;
            object[] datasourceattributes = _context.TestMethodInfo.GetCustomAttributes(typeof (EnumurableDataSourceAttribute), false);
            Type getclasstype = testMethodAssembly.GetType(((EnumurableDataSourceAttribute)datasourceattributes[0]).ClassName);
            MethodInfo getmethodforobjects = getclasstype.GetMethod(((EnumurableDataSourceAttribute) datasourceattributes[0]).DataSourceName);
            Use the line below if there are parameters that needs to be passed to the method. 
            ParameterInfo[] methodparameters = getmethodforobjects.GetParameters();
            To instantiate a new concreate class
            object classInstance = Activator.CreateInstance(getclasstype, null);
            Invoke(null,null) = The first null parameter specifies whether it's a static class or not. For static, leave it null
            IEnumerable<object> enmeruableobjects = getmethodforobjects.Invoke(classInstance, null) as IEnumerable<object>;
            IEnumerable<object> enmeruableobjects = getmethodforobjects.Invoke(null, null) as IEnumerable<object>;
            var testresults = new TestResults();
            //This is where each object will be enumarated for the test method. 
            foreach (var obj in enmeruableobjects)
                testresults.AddTestResult(_context.InnerInvoker.Invoke(obj), new object[1] { obj });
            var output = testresults.GetAllResults();
            return output;

Test Project Setup:

Once, you’ve successfully build your assembly project (custom TestClass attribute), you need to register the custom extension class in your local machine. This is a custom test assembly/adapter so, we’ll need to:

  • Make changes to the registry
  • Add the compiled assembly in the install directory for your VS version. In my case, I’m using Visual Studio 2015 so your custom assemblies will be copied to:  C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\PrivateAssemblies

Luckily, there’s a batch script that lets you do all of these steps. The only thing you need to do is:

  • Change the version of VS to your working VS edition
  • Change the assembly namespace and class reference

The deployment script: (You can also download the deployment script from this blog. Scroll at the bottom of the blog post)

@echo off
:: Install a MSTest unit test type extension
:: which defines a new test class attribute
:: and how to execute its test methods and
:: interpret results.
:: NOTE: Only VS needs this and the registration done; the xcopyable mstest uses
:: TestTools.xml virtualized registry file updated which we already have done in sd


:: All the files we need to copy or register are realtive to this script folder
set extdir=%~dp0

:: Get 32 or 64-bit OS
set win64=0
if not "%ProgramFiles(x86)%" == "" set win64=1
if %win64% == 1 (
    set vs14Key=HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0
) else (
    set vs14Key=HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\14.0

:: Get the VS installaton path from the Registry
for /f "tokens=2*" %%i in ('reg.exe query %vs14Key% /v InstallDir') do set vsinstalldir=%%j

:: Display some info
echo =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-
echo Please ensure that you are running with adminstrator privileges
echo to copy into the Visual Studio installation folder add keys to the Registry.
echo Any access denied messages probably means you are not.
echo =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-
echo 64-bit OS: %win64%
echo Visual Studio 14.0 regkey:   %vs14Key%
echo Visual Studio 14.0 IDE dir:  %vsinstalldir%

:: Copy the SSM test type extension assembly to the VS private assemblies folder

set extdll=AAG.Test.Core.CustomTestExtenstions.dll
set vsprivate=%vsinstalldir%PrivateAssemblies
echo Copying to VS PrivateAssemblies: %vsprivate%\%extdll%
copy /Y %extdir%%extdll% "%vsprivate%\%extdll%"

:: Register the extension with mstest as a known test type
:: (SSM has two currently, both are in the same assembly)

echo Registering the unit test types extensions for use in VS' MSTest

:: Keys Only for 64-bit
if %win64% == 1 (
    set vs14ExtKey64=HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\TestTypeExtensions
    set vs14_configExtKey64=HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0_Config\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\TestTypeExtensions

:: Keys for both 32 and 64-bit
set vs14ExtKey=HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\14.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\TestTypeExtensions
set vs14_configExtKey=HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\14.0_Config\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\TestTypeExtensions

:: Register the TestClassCollectionAttribute
set regAttrName=TestClassCollectionAttribute
set regProvider="AAG.Test.Core.CustomTestExtenstions.TestClassCollectionAttribute, AAG.Test.Core.CustomTestExtenstions"
if  %win64% == 1 (
    reg add %vs14ExtKey64%\%regAttrName%        /f /v AttributeProvider /d %regProvider%
    reg add %vs14_ConfigExtKey64%\%regAttrName% /f /v AttributeProvider /d %regProvider%
reg add %vs14ExtKey%\%regAttrName%        /f /v AttributeProvider /d %regProvider%
reg add %vs14_ConfigExtKey%\%regAttrName% /f /v AttributeProvider /d %regProvider%

exit /b %errorlevel%

Creating the tests in VS:

In your test project, add a reference to the custom MSTest Assemblies


In your tests, make sure to use the custom test class and enumerator attribute that we previously defined. Here’s a sample of these test methods that uses different IEnumerable objects.

    public class MethodCollectionTests

        public TestContext TestContext { get; set; }

        public void TestInit()

        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get5Employees")]
        public void Verify5Employees(Employee employee)
            Console.WriteLine($"Employee FirstName: {employee.Displayname}");
            TestContext.WriteLine($"Test Case Passed for {TestContext.TestName} with Data: {employee.Displayname}");

        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get20Employees")]
        public void Verify20Employees(Employee employee)
            Console.WriteLine($"Employee FirstName: {employee.Displayname}");
            TestContext.WriteLine($"Test Case Passed for {TestContext.TestName} with Data: {employee.Displayname}");

        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get5Cars")]
        public void Verify5Cars(Car car)
            Console.WriteLine($"Car Info: Type: {car.CarType} Cost: {car.Cost.ToString("C")}");
            TestContext.WriteLine($"Test Case Passed for {TestContext.TestName} with Data: {car.CarType} with Id: {car.Id}");

        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get13Cars")]
        public void Verify13Cars(Car car)
            Console.WriteLine($"Car Info: Type: {car.CarType} Cost: {car.Cost.ToString("C")}");
            TestContext.WriteLine($"Test Case Passed for {TestContext.TestName} with Data: {car.CarType} with Id: {car.Id}");

        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get5EmployeesWithCars")]
        public void Verify5EmployeesWithCars(EmployeeWithCar employeeWithCar)
            TestContext.WriteLine($"Test Case Passed for {TestContext.TestName} with Data: Name: {employeeWithCar.Employee.Displayname} Car: {employeeWithCar.Car.CarType} with Id: {employeeWithCar.Employee.Id}");

        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get10SequentialInts")]
        public void Verify10SequentialInts(int intcurrent)
            Assert.IsInstanceOfType(intcurrent, typeof(int));
            TestContext.WriteLine($"Current int Value: {intcurrent}");

        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get5StringObjects")]
        public void VerifyGet5StringObjects(string stringcurrent)
            Assert.IsInstanceOfType(stringcurrent, typeof(string));
            TestContext.WriteLine($"Current string Value: {stringcurrent}");

The Helper class defines the helper methods to generate test data:

public static class Helper
        public static IEnumerable<object> Get5Employees()
            var employees = GenerateData.GetEmployees(5);
            return (IEnumerable<object>) employees;

        public static IEnumerable<object> Get20Employees()
            var employees = GenerateData.GetEmployees(20);
            return (IEnumerable<object>) employees;
        public static IEnumerable<object> Get5Cars()
            var cars = GenerateData.GetCars(5);
            return (IEnumerable<object>) cars;

        public static IEnumerable<object> Get13Cars()
            var cars = GenerateData.GetCars(13);
            return (IEnumerable<object>)cars;

        public static IEnumerable<object> Get5EmployeesWithCars()
            var cars = GenerateData.GetCars(5);
            var employees = GenerateData.GetEmployees(5);
            var employeeswithcars = new List<EmployeeWithCar>();
            for (int i = 0; i < cars.Count; i++)
                var employeewithcar = new EmployeeWithCar
                    Car = cars[i],
                    Employee = employees[i]
            return (IEnumerable<object>)employeeswithcars;

        public static IEnumerable<object> Get10SequentialInts()
            var intobjects = new int[] {1, 2, 3, 4, 5,6,7,8,9, 10};
            return (IEnumerable<object>) intobjects.Cast<object>();

        public static IEnumerable<object> Get5StringObjects()
            var stringobjects = new string[] {"String1", "String2", "String3", "String4", "String5" };
            return (IEnumerable<object>)stringobjects.Cast<object>();

The Execution results!


And the output for each of the result!