Validating and Unit Testing Web API (2) Route Attribute Parameters

Personally, I like to isolate business rules and/or validations outside of MVC Controllers. In this case, API Controllers. I use ActionFilterAttribute to define my checks on parameters being passed in my MVC Web API routes.

Here’s an example of a WebAPI route with parameter binding:

// GET: /1/employees/AA0000111"
[Route("{WebServiceVersion}/employees/{employeeId}")]
[ValidateEmployeeId]
        public IHttpActionResult GetUser(string employeeid, int WebServiceVersion = 1)
        {
            // GET: Do something with webServiceVersion value like logging.
            var user = _emprepository.GetUser(employeeid);
            return Content(HttpStatusCode.OK, user);
        }

I want to isolate validating employeeid outside of my controller for a couple of reasons:

1) Isolation – You may have multiple cases on validating your parameters. In this case, employeeId can be permutated in different ways specially because it is a string. Other developers can easily get lost on what the action controller is actually doing if you have long code that includes all various validations

2) Good development practice – I prefer to see nice clean code and separation on what my controllers do vs business rules

3) Testing – I can isolate testing on my controllers vs business rules. This is really the motivating factor for me.

That said, let’s take a look at the ActionFilterAttribute further. For more information on this, see:

(NOTE: There are 2 versions of ActionFilterAttribute)

System.Web.Http.Filters

System.Web.Mvc

When unit testing, make sure you’re writing the correct tests for your filter. In this case, I’m using the namespace: System.Web.Http.Filters

public class ValidateEmployeeIdAttribute : ActionFilterAttribute
    {
        public override void OnActionExecuting(HttpActionContext actionContext)
        {
            var employeeid = actionContext.ActionArguments["employeeid"].ToString();
            if (string.IsNullOrEmpty(employeeid) || employeeid.ToLower() == "<somecheck>" ||
                employeeid.ToLower() == "<replace and use other validation such as regex>")
            {
                actionContext.Response = actionContext.Request.CreateResponse(HttpStatusCode.BadRequest,
                    $"Input parameter error, employeeId: {employeeid} -  not specified, null or bad format",
                    actionContext.ControllerContext.Configuration.Formatters.JsonFormatter);
            }
            base.OnActionExecuting(actionContext);
        }
    }

Note in the preceding code for the controller that I decorated the web api action method with: [ValidateEmployeeId]

This instruct the controller to use the custom ActionFilterAttribute that I created above

Testing your custom validate via UNIT Test/s:

For simplicity, I used MSTest that comes with visual studio.

[TestMethod, TestCategory("UserController")]
        public void Validate_EmpId_ActionFilterAttribute()
        {
            var mockactioncontext = new HttpActionContext
            {
                ControllerContext = new HttpControllerContext
                {
                    Request = new HttpRequestMessage()
                },
                ActionArguments = { { "employeeid", "<somecheck>" } }
            };

            mockactioncontext.ControllerContext.Configuration = new HttpConfiguration();
            mockactioncontext.ControllerContext.Configuration.Formatters.Add(new JsonMediaTypeFormatter());
            
            var filter = new ValidateEmployeeIdAttribute();
            filter.OnActionExecuting(mockactioncontext);
            Assert.IsTrue(mockactioncontext.Response.StatusCode == HttpStatusCode.BadRequest);
        }

At this point, you should have separation of code to validate your “validations” vs controller.

Using fiddler, I can see that whenever I submit a request that has an invalid value for employeeid, I get the correct response:

fiddlertrace

Using XML Data Transform (XDT) to automatically configure app.config during Nuget Package Install

This should be fairly straight forward as mentioned on nuget.org’s site right? Well, not quite. I’ve spent some time reading through the blog posts and it’s not quite straightforward. Hopefully this post is the simplified version. In my case, the scenario is simply to add entries in the appSettings key node within the app.config file. Nuget.org’s site has the following docs:

Configuration File and Source Code Transformations

https://docs.nuget.org/create/configuration-file-and-source-code-transformations

How to use XDT in NuGet – Examples and Facts

http://blog.nuget.org/20130920/how-to-use-nugets-xdt-feature-examples-and-facts.html

The steps below will hopefully guide you through the initial steps to get your app.config (or web.config) files to be modified during and after installing your nuget packages. After which you can look at all different XDT transformation processes in the following doc:

Web.config Transformation Syntax for Web Project Deployment Using Visual Studio

https://msdn.microsoft.com/en-us/library/dd465326(v=vs.110).aspx

Step 1: Create both app.config.install.xdt and app.config.uninstall.xdt

From Nuget site: “Starting with NuGet 2.6, XML-Document-Transform (XDT) is supported to transform XML files inside a project. The XDT syntax can be utilized in the .install.xdt and .uninstall.xdt file(s) under the package’s Content folder, which will be applied during package installation and uninstallation time, respectively.”

The location of these files don’t quite matter. If these files are located in the same directory as where you have your assemblies for nuget package, even better. You’ll need to reference these 2 files as “content” folder locations in the .nuspec file. Nuspec file is the blue print for creating your nuget package.

app.config.install.xdt

<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings xdt:Transform="InsertIfMissing">
    </appSettings>
  <appSettings>
    <add key="Key1" xdt:Transform="Remove" xdt:Locator="Match(key)" />
    <add key="Key1" value="Value1" xdt:Transform="Insert"/>
    <add key="Key2" xdt:Transform="Remove" xdt:Locator="Match(key)"/>
    <add key="Key2" value="Value2" xdt:Transform="Insert" />
  </appSettings>
</configuration>

Let’s break this down. There are 2 appSettings node in this xml file. One to check if the appSettings node exist (InsertIfMissing) and the 2nd, if it does exist, it will remove the key value pair matching the keyword and then add it again. Why do this 2 step process? This is to ensure that you will only have one entry per key. However, you could probably get away using InsertIfMissing as well.

app.config.uninstall.xdt

<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings xdt:Transform="InsertIfMissing">
    </appSettings>
  <appSettings>
    <add key="Key1" xdt:Transform="Remove" xdt:Locator="Match(key)" />
    <add key="Key2" xdt:Transform="Remove" xdt:Locator="Match(key)"/>
 </appSettings>
</configuration>
The uninstall file is pretty straightforward. Remove the app setting keys if they exist. Although, in this case, I’m not deleting the appSettings node. Leaving the appSettings node in your config file will not cause any issues.

Step 2: Modify your nuspec file to include both the .install.xdt and .uninstall.xdt file(s) as content folders.

.nuspec file is the core or blue print for generating your nuget package. Here’s an example of a .nuspec file. For more information, go here: http://docs.nuget.org/Create/Nuspec-Reference

In this example, you’ll need to refer for both .install.xdt and .uninstall.xdt file(s) as target content folders:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2011/08/nuspec.xsd">
  <metadata>
    <id>Package1</id>
    <version>1.1</version>
    <title>Nuget Package 1</title>
    <authors>QE Dev</authors>
    <owners>Don Tan</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>Package 1 Testing</description>
<summary>Application Config change</summary>

    <releaseNotes>
      - Support for Application Config change
    </releaseNotes>
    <copyright>Copy Right</copyright>
    <language>en-US</language>
    <dependencies>
      <dependency id="Microsoft.ApplicationInsights" version="2.1.0" />
    </dependencies>
    <references>
      <reference file="Package1.dll" />
    </references>
  </metadata>
  <files>
    <file src="Package1.dll" target="lib\net45\Package1.dll.dll" />
    <!--Add Section to Uninstall and Re-install Application.Config files-->
    <file src="app.config.install.xdt" target="content" />
    <file src="app.config.uninstall.xdt" target="content" />
  </files>
</package>

Step 3: Test the generated nuget package and verify if your application config (app.config) settings have been modified

Custom Build Triggers in VSTS

In my previous posts, I’ve shown people how to use VSTS (formerly known as VSO) to trigger continuous testing using builds and release management. I was able to utilize new reporting capabilities in build, particularly, test reports. I created reports that shows pass/fail trends for tests in my build definitions.

PassFailTrend

There are still limitations (or in this case features I wish Microsoft would consider such as customizing test reports from builds as well as showing pass/fail trends past 10 builds). My biggest disappointment thus far is “NOT” able to schedule build (with tests) using re-occurring pattern/s. As of writing this post, you can schedule builds in VSTS however, you have to “manually” keep adding scheduled times.

Scheduled

Imagine a scenario where you need to run a build every hour (or half hour), you have to manually add new times every hour, in this case, 24 times. Very inconvenient.

Fortunately, VSTS has public API’s that allows us to access build execution and trigger. With the public API’s I was able to write a very simple console app and use Windows’ built in “Task Scheduler” functionality. One would say, why not create a windows services? Yes, that’s option but I would make a point back to say: “Why develop a windows service further complicating the process where Windows has ‘Task Scheduler’ that’s been tested and used more broadly?”

Below is the code:

NOTE: You need to refer to the following Nuget Packages:

  • Microsoft.TeamFoundationServer.ExtendedClient
  • Microsoft.TeamFoundationServer.Client
  • Microsoft.VisualStudio.Services.Client
  • Microsoft.VisualStudio.Services.InteractiveClient
static class Program
    {
        static void Main(string[] args)
        {
            var buildoutputmodel = SetupBuildOutputModel();
            var vssconnection = new VssConnection(
                new Uri(buildoutputmodel.VsoUrl),
                new VssBasicCredential(buildoutputmodel.UserName, buildoutputmodel.Password)
                );
            var buildHttpClient = vssconnection.GetClient<BuildHttpClient>();
            //Below is my implementation of triggering multiple builds. I simply used the app.config to specify the build's ID, split each entry and validate. 
            ConfigurationManager.AppSettings["builddefinitionids"].Split(',').ToList().ForEach(
                buildid =>
                {
                    string stringoutput;
                    try
                    {
                        var id = buildid.ValidateBuildId();
                        DefinitionReference definitionReference = new DefinitionReference
                        {
                            Id = id,
                            Project = new TeamProjectReference
                            {
                                Name = buildoutputmodel.TeamProjectName
                            }
                        };
                        var build = new Build { Definition = definitionReference };
                        //This is where you trigger the build
                        var buildnumber = buildHttpClient.QueueBuildAsync(build,
                            buildoutputmodel.TeamProjectName).Result;
                        stringoutput = $"Build Triggered... \nBuild Number: {buildnumber} \nBuild Definition ID: {definitionReference.Id} \nTeam Project: {definitionReference.Project.Name}\n";
                        Console.WriteLine(stringoutput);
                        AsLogger.Info(stringoutput);
                    }
                    catch (Exception ex)
                    {
                        stringoutput = $"Exception Occurred: \n{ex.Message} \n{ex.InnerException}\n";
                        Console.WriteLine(stringoutput);
                        AsLogger.Error(stringoutput);
                    }
                });
        }

        private static BuildOutputModel SetupBuildOutputModel()
        {
            return new BuildOutputModel
            {
                UserName = ConfigurationManager.AppSettings["username"],
                Password = ConfigurationManager.AppSettings["password"],
                VsoUrl = ConfigurationManager.AppSettings["vsourl"],
                TeamProjectName = ConfigurationManager.AppSettings["teamproject"],
                BuilDefinitionName = ConfigurationManager.AppSettings["builddefinition"],
                GitRepo = ConfigurationManager.AppSettings["gitrepo"]
            };
        }
    }

Once you compile the code (.exe), simply create a scheduled task using Windows’ Task Scheduler:

TaskScheduler

Then the execution:

VSTSQueue

Working with VSTS Rest APIs

I’ve been working with VSTS for quite some time now and wanted to share some of the sample code I’ve written to work with VSTS data. As I work with many teams, there have been requests such as getting specific metadata during and/or after build. Examples would be people wanting to get specific data from associated work-items during builds or collection level licensing information for your users. Here are I’ll tap in to specific areas:

VSTS Builds, VSTS Work-Items, VSTS GIT (Commits), VSTS User License Information

You’ll need to the following Nuget Packages:

  • Microsoft.TeamFoundationServer.ExtendedClient
  • Microsoft.TeamFoundationServer.Client
  • Microsoft.VisualStudio.Services.Client
  • Microsoft.VisualStudio.Services.InteractiveClient

Sample Code to retrieve all Builds from a given VSTS Team Project Build Definition containing associated commits and work-items:

using System;
using System.Configuration;
using System.Linq;
using System.Text;
using AAG.Test.Core.Logger;
using Microsoft.TeamFoundation.Build.WebApi;
using Microsoft.TeamFoundation.SourceControl.WebApi;
using Microsoft.VisualStudio.Services.Client;
using VSTSApi.Entities;
using Microsoft.VisualStudio.Services.Common;
using Microsoft.TeamFoundation.WorkItemTracking.WebApi;

namespace VSTSApi
{
    class Program
    {
        private static BuildOutputModel _buildoutputmodel;

        private static string VssAccountUrl { get; set; } = ConfigurationManager.AppSettings["VssAccountUrl"];

        static void Main(string[] args)
        {
            try
            {
                StringBuilder outputStringBuilder = new StringBuilder();
                var buildoutputmodel = SetBuildOutputModel(args);
                var creds = new VssClientCredentials(false);
                creds.PromptType = CredentialPromptType.PromptIfNeeded;
                var vssConnection = new VssConnection(new Uri(buildoutputmodel.VSOUrl + "/defaultcollection"), new VssBasicCredential(buildoutputmodel.UserName, buildoutputmodel.Password));

                var buildserver = vssConnection.GetClient<BuildHttpClient>();
                var workitems = vssConnection.GetClient<WorkItemTrackingHttpClient>();
                var commititems = vssConnection.GetClient<GitHttpClient>();
                var builds = buildserver.GetBuildsAsync(_buildoutputmodel.TeamProjectName).Result;
                var targetbuilds = builds.Where(definition => definition.Definition.Name.Contains(buildoutputmodel.BuilDefinitionName));
                foreach (var build in targetbuilds)
                {
                    outputStringBuilder.AppendLine($"Name: {build.Definition.Name} : BuildID: {build.Id}");
                    var associatedcommits = buildserver.GetBuildCommitsAsync(build.Definition.Project.Name,
                        build.Id).Result;
                    if (associatedcommits.Any())
                        outputStringBuilder.AppendLine($"All Commits Made for this Build:  {Environment.NewLine} ========= {Environment.NewLine} ");
                    associatedcommits.ForEach(commit =>
                    {
                        var user = commititems.GetCommitAsync(buildoutputmodel.TeamProjectName, commit.Id, buildoutputmodel.GitRepo).Result.Author;
                        outputStringBuilder.AppendLine($"ID: {commit.Id} Committed By: {user.Name}  E-mail: {user.Email} {Environment.NewLine} Description: {commit.Message} {Environment.NewLine}");
                    });
                    var commits = associatedcommits.Select(change => change.Id);
                    var associatedworkitems = buildserver.GetBuildWorkItemsRefsAsync(commits,
                        build.Definition.Project.Name, build.Id).Result;
                    if (associatedworkitems.Any())
                        outputStringBuilder.AppendLine($"All Associated Workitems for this Build:  {Environment.NewLine} ========= {Environment.NewLine} ");
                    foreach (var wi in associatedworkitems)
                    {
                        outputStringBuilder.AppendLine($"ID : {wi.Id} : URL: {wi.Url}");
                        var workitem = workitems.GetWorkItemAsync(int.Parse(wi.Id)).Result;
                        outputStringBuilder.AppendLine($"Title : {workitem.Fields["Title"]} : Description: {workitem.Fields["Description"]}");
                    }
                    outputStringBuilder.AppendLine($"{Environment.NewLine} ========= {Environment.NewLine} ");
                }
                //}

                DumpData(outputStringBuilder.ToString(), Console.WriteLine);
                DumpData(outputStringBuilder.ToString(), print => AsLogger.Info(print));
                Console.WriteLine("Press Any Key to Continue...");
                Console.ReadKey();
            }
            catch (Exception exception)
            {
                throw new Exception($"Error with Application: {exception.Message}", exception.InnerException);
            }
        }

        static void DumpData(string stringoutput, Action<string> print)
        {
            print(stringoutput);
        }

        static BuildOutputModel SetBuildOutputModel(string[] args)
        {
            _buildoutputmodel = new BuildOutputModel
            {
                UserName = ConfigurationManager.AppSettings["username"],
                Password = ConfigurationManager.AppSettings["password"],
                VSOUrl = ConfigurationManager.AppSettings["vsourl"],
                TeamProjectName = ConfigurationManager.AppSettings["teamproject"],
                BuilDefinitionName = ConfigurationManager.AppSettings["builddefinition"],
                GitRepo = ConfigurationManager.AppSettings["gitrepo"]
            };
            return _buildoutputmodel;
        }
    }
}

Entity

namespace VSTSApi.Entities
{
    public class BuildOutputModel
    {
        public string UserName { get; set; }

        public string Password { get; set; }

        public string BuilDefinitionName { get; set; }

        public string TeamProjectName { get; set; }

        public string VSOUrl { get; set; }

        public string GitRepo { get; set; }
    }
}

Sample Code for Getting User Information/Licenses:

using System;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using VSTSAccountAdmin.Model;
using Microsoft.VisualStudio.Services.Client;
using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.Identity.Client;
using Microsoft.VisualStudio.Services.Licensing;
using Microsoft.VisualStudio.Services.Licensing.Client;

namespace VSTSAccountAdmin
{
    public class Program
    {
        private static string VssAccountUrl { get; set; } = ConfigurationManager.AppSettings["VssAccountUrl"];
        private static string VssAccountName { get; set; }
        private static License VssLicense { get; set; }

        private static List<VSOUserInfo> _vsousers;



        public static void Main(string[] args)
        {
            try
            {
                _vsousers = new List<VSOUserInfo>();

                // Create a connection to the specified account.
                // If you change the false to true, your credentials will be saved.
                var creds = new VssClientCredentials(false);
                creds.PromptType = CredentialPromptType.PromptIfNeeded;
                var vssConnection = new VssConnection(new Uri(VssAccountUrl), creds);

                // We need the clients for tw4o services: Licensing and Identity
                var licensingClient = vssConnection.GetClient<LicensingHttpClient>();
                var identityClient = vssConnection.GetClient<IdentityHttpClient>();

                var entitlements = licensingClient.GetAccountEntitlementsAsync().Result;
                IEnumerable<AccountEntitlement> accountEntitlements = entitlements as IList<AccountEntitlement> ??
                                                                      entitlements.ToList();
                var userIds = accountEntitlements.Select(entitlement => entitlement.UserId).ToList();
                var users = identityClient.ReadIdentitiesAsync(userIds).Result.ToDictionary(item => item.Id);
                foreach (var entitlement in accountEntitlements)
                {
                    var user = users[entitlement.UserId];
                    _vsousers.Add(new VSOUserInfo()
                    {
                        DisplayName = user.DisplayName,
                        LastAccessDate = entitlement.LastAccessedDate,
                        License = entitlement.License.ToString().ToLowerInvariant(),
                        UserID = entitlement.UserId
                    });
                    var stringoutput =
                        $"{Environment.NewLine}Name: {user.DisplayName}, UserId: {entitlement.UserId}, License: {entitlement.License}.";
                    Console.WriteLine(stringoutput);
                }
            }
            catch (Exception ex)
            {
                throw new ArgumentException(ex.Message, ex.InnerException);
            }

        }
    }
}

Entity:

namespace VSTSAccountAdmin.Model
{
    public class VSOUserInfo
    {
        public string DisplayName { get; set; }

        public Guid UserID { get; set; }

        public string License { get; set; }

        public DateTimeOffset LastAccessDate { get; set; }

    }
}

MSTEST: Extending Data Driven Tests to use IENUMERABLE<Object> as the Data Source

One great feature that I like in NUnit is the capability to use collection types for data driven tests. Meaning, you don’t have to open up an external data source connection, pull data and use it to drive parameters for your tests. With a simple attribute in NUnit, you can drive tests as indicated here:

TestCaseSourceAttribute

http://www.nunit.org/index.php?p=testCaseSource&r=2.5.3

NUnit implementation allows you to enumerate from a collection to data drive your tests. MSTest has the same extensibility and is outlined in the following blog:

Extending the Visual Studio Unit Test Type

http://blogs.msdn.com/b/vstsqualitytools/archive/2009/09/04/extending-the-visual-studio-unit-test-type-part-1.aspx

From this blog, I was able to go through the steps and process how MSTest invokes and passes objects in a test method. When MSTest executes, the flow goes through:

  1. TestClassExtensionAttribute calls : GetExecution()
  2. TestExtensionExecution calls : CreateTestMethodInvoker(TestMethodInvokerContext context)
  3. ITestMethodInvoker calls : Invoke(params object[] parameters)

image

Throughout this process, you can use custom attributes and utilize attribute properties for passing in test data. Its best that you use custom attributes in the ITestMethodInvoker.Invoke()

In my solution, I want to develop a fast way of invoking IEnumerable<object> as my test data. In this case, I’ll consume custom attributes to provide a classname and methodname to return test data through reflection. I’ll then use that in ITestMethodInvoker.Invoke() to enumerate objects for my tests.

  • ClassName: class holding the method to generate test data
  • DataSourceName: method within the class that generates any test data

Project Setup:

Make sure that you have the following references in your project:

  • Microsoft.VisualStudio.QualityTools.Common.dll
  • Microsoft.VisualStudio.QualityTools.UnitTestFramework
  • Microsoft.VisualStudio.QualityTools.Vsip.dll

These assemblies are included as part of the .Net framework. Simply browse in the references section in your project

The Custom Attribute:

    [global::System.AttributeUsage(AttributeTargets.Method, Inherited = false, AllowMultiple = false)]
    public class EnumurableDataSourceAttribute : Attribute
    {
        public string DataSourceName { get; set; }
        public string ClassName { get; set; }


        public EnumurableDataSourceAttribute(string className, string dataSourceName)
        {
            this.DataSourceName = dataSourceName;
            this.ClassName = className;
        }
    } 

Test Class Implementation:

[Serializable]
    public class TestClassCollectionAttribute : TestClassExtensionAttribute
    {
        public override Uri ExtensionId => new Uri("urn:TestClassAttribute");

        public override object GetClientSide()
        {
            return base.GetClientSide();
        }

        public override TestExtensionExecution GetExecution()
        {
            return new TestExtension();
        }
    }

Test Extension:

public class TestExtension : TestExtensionExecution
    {
        public override void Initialize(TestExecution execution)
        {
            
        }

        public override ITestMethodInvoker CreateTestMethodInvoker(TestMethodInvokerContext context)
        {
            return new TestInvokerMethodCollection(context);
        }

        public override void Dispose()
        {
            
        }
    }

Test Method Invoker:

public class TestInvokerMethodCollection : ITestMethodInvoker
    {
        private readonly TestMethodInvokerContext _context;
        
        public TestInvokerMethodCollection(TestMethodInvokerContext context)
        {
            Debug.Assert(context != null);
            _context = context;
        }
        public TestMethodInvokerResult Invoke(params object[] parameters)
        {
            Trace.WriteLine($"Begin Invoke:Test Method Name: {_context.TestMethodInfo.Name}");
            Assembly testMethodAssembly = _context.TestMethodInfo.DeclaringType.Assembly;
            object[] datasourceattributes = _context.TestMethodInfo.GetCustomAttributes(typeof (EnumurableDataSourceAttribute), false);
            Type getclasstype = testMethodAssembly.GetType(((EnumurableDataSourceAttribute)datasourceattributes[0]).ClassName);
            MethodInfo getmethodforobjects = getclasstype.GetMethod(((EnumurableDataSourceAttribute) datasourceattributes[0]).DataSourceName);
            /*
            Use the line below if there are parameters that needs to be passed to the method. 
            ParameterInfo[] methodparameters = getmethodforobjects.GetParameters();
            To instantiate a new concreate class
            object classInstance = Activator.CreateInstance(getclasstype, null);
            Invoke(null,null) = The first null parameter specifies whether it's a static class or not. For static, leave it null
            IEnumerable<object> enmeruableobjects = getmethodforobjects.Invoke(classInstance, null) as IEnumerable<object>;
            */
            IEnumerable<object> enmeruableobjects = getmethodforobjects.Invoke(null, null) as IEnumerable<object>;
            var testresults = new TestResults();
            //This is where each object will be enumarated for the test method. 
            foreach (var obj in enmeruableobjects)
            {
                testresults.AddTestResult(_context.InnerInvoker.Invoke(obj), new object[1] { obj });
            }           
            var output = testresults.GetAllResults();
            _context.TestContext.WriteLine(output.ExtensionResult.ToString());
            return output;
        }
    }

Test Project Setup:

Once, you’ve successfully build your assembly project (custom TestClass attribute), you need to register the custom extension class in your local machine. This is a custom test assembly/adapter so, we’ll need to:

  • Make changes to the registry
  • Add the compiled assembly in the install directory for your VS version. In my case, I’m using Visual Studio 2015 so your custom assemblies will be copied to:  C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\PrivateAssemblies

Luckily, there’s a batch script that lets you do all of these steps. The only thing you need to do is:

  • Change the version of VS to your working VS edition
  • Change the assembly namespace and class reference

The deployment script: (You can also download the deployment script from this blog. Scroll at the bottom of the blog post)

http://blogs.msdn.com/b/qingsongyao/archive/2012/03/28/examples-of-mstest-extension.aspx?CommentPosted=true

@echo off
::------------------------------------------------
:: Install a MSTest unit test type extension
:: which defines a new test class attribute
:: and how to execute its test methods and
:: interpret results.
::
:: NOTE: Only VS needs this and the registration done; the xcopyable mstest uses
:: TestTools.xml virtualized registry file updated which we already have done in sd
::------------------------------------------------

setlocal

:: All the files we need to copy or register are realtive to this script folder
set extdir=%~dp0

:: Get 32 or 64-bit OS
set win64=0
if not "%ProgramFiles(x86)%" == "" set win64=1
if %win64% == 1 (
    set vs14Key=HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0
) else (
    set vs14Key=HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\14.0
)

:: Get the VS installaton path from the Registry
for /f "tokens=2*" %%i in ('reg.exe query %vs14Key% /v InstallDir') do set vsinstalldir=%%j


:: Display some info
echo.
echo =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-
echo Please ensure that you are running with adminstrator privileges
echo to copy into the Visual Studio installation folder add keys to the Registry.
echo Any access denied messages probably means you are not.
echo =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-
echo.
echo 64-bit OS: %win64%
echo Visual Studio 14.0 regkey:   %vs14Key%
echo Visual Studio 14.0 IDE dir:  %vsinstalldir%

::
:: Copy the SSM test type extension assembly to the VS private assemblies folder
::

set extdll=AAG.Test.Core.CustomTestExtenstions.dll
set vsprivate=%vsinstalldir%PrivateAssemblies
echo Copying to VS PrivateAssemblies: %vsprivate%\%extdll%
copy /Y %extdir%%extdll% "%vsprivate%\%extdll%"

::
:: Register the extension with mstest as a known test type
:: (SSM has two currently, both are in the same assembly)
::

echo Registering the unit test types extensions for use in VS' MSTest

:: Keys Only for 64-bit
if %win64% == 1 (
    set vs14ExtKey64=HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\TestTypeExtensions
    set vs14_configExtKey64=HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0_Config\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\TestTypeExtensions
)

:: Keys for both 32 and 64-bit
set vs14ExtKey=HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\14.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\TestTypeExtensions
set vs14_configExtKey=HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\14.0_Config\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\TestTypeExtensions

:: Register the TestClassCollectionAttribute
set regAttrName=TestClassCollectionAttribute
set regProvider="AAG.Test.Core.CustomTestExtenstions.TestClassCollectionAttribute, AAG.Test.Core.CustomTestExtenstions"
if  %win64% == 1 (
    reg add %vs14ExtKey64%\%regAttrName%        /f /v AttributeProvider /d %regProvider%
    reg add %vs14_ConfigExtKey64%\%regAttrName% /f /v AttributeProvider /d %regProvider%
)
reg add %vs14ExtKey%\%regAttrName%        /f /v AttributeProvider /d %regProvider%
reg add %vs14_ConfigExtKey%\%regAttrName% /f /v AttributeProvider /d %regProvider%

:eof
endlocal
exit /b %errorlevel%

Creating the tests in VS:

In your test project, add a reference to the custom MSTest Assemblies

image

In your tests, make sure to use the custom test class and enumerator attribute that we previously defined. Here’s a sample of these test methods that uses different IEnumerable objects.

[TestClassCollection]
    public class MethodCollectionTests
    {

        public TestContext TestContext { get; set; }

        [TestInitialize()]
        public void TestInit()
        {
        }

        [TestMethod]
        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get5Employees")]
        public void Verify5Employees(Employee employee)
        {
            Assert.IsFalse(String.IsNullOrEmpty(employee.Displayname));
            Console.WriteLine($"Employee FirstName: {employee.Displayname}");
            TestContext.WriteLine($"Test Case Passed for {TestContext.TestName} with Data: {employee.Displayname}");
        }

        [TestMethod]
        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get20Employees")]
        public void Verify20Employees(Employee employee)
        {
            Assert.IsFalse(String.IsNullOrEmpty(employee.Displayname));
            Console.WriteLine($"Employee FirstName: {employee.Displayname}");
            TestContext.WriteLine($"Test Case Passed for {TestContext.TestName} with Data: {employee.Displayname}");
        }

        [TestMethod]
        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get5Cars")]
        public void Verify5Cars(Car car)
        {
            Assert.IsNotNull(car);
            Assert.IsFalse(String.IsNullOrEmpty(car.Description));
            Console.WriteLine($"Car Info: Type: {car.CarType} Cost: {car.Cost.ToString("C")}");
            TestContext.WriteLine($"Test Case Passed for {TestContext.TestName} with Data: {car.CarType} with Id: {car.Id}");
        }

        [TestMethod]
        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get13Cars")]
        public void Verify13Cars(Car car)
        {
            Assert.IsNotNull(car);
            Assert.IsFalse(String.IsNullOrEmpty(car.Description));
            Console.WriteLine($"Car Info: Type: {car.CarType} Cost: {car.Cost.ToString("C")}");
            TestContext.WriteLine($"Test Case Passed for {TestContext.TestName} with Data: {car.CarType} with Id: {car.Id}");
        }

        [TestMethod]
        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get5EmployeesWithCars")]
        public void Verify5EmployeesWithCars(EmployeeWithCar employeeWithCar)
        {
            Assert.IsNotNull(employeeWithCar);
            Assert.IsFalse(String.IsNullOrEmpty(employeeWithCar.Id));
            Assert.IsNotNull(employeeWithCar.Car);
            Assert.IsNotNull(employeeWithCar.Employee);
            TestContext.WriteLine($"Test Case Passed for {TestContext.TestName} with Data: Name: {employeeWithCar.Employee.Displayname} Car: {employeeWithCar.Car.CarType} with Id: {employeeWithCar.Employee.Id}");
        }

        [TestMethod]
        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get10SequentialInts")]
        public void Verify10SequentialInts(int intcurrent)
        {
            Assert.IsInstanceOfType(intcurrent, typeof(int));
            TestContext.WriteLine($"Current int Value: {intcurrent}");
        }

        [TestMethod]
        [EnumurableDataSourceAttribute("CustomTestExtenstions.Tests.Helper.Helper", "Get5StringObjects")]
        public void VerifyGet5StringObjects(string stringcurrent)
        {
            Assert.IsInstanceOfType(stringcurrent, typeof(string));
            TestContext.WriteLine($"Current string Value: {stringcurrent}");
        }
    }

The Helper class defines the helper methods to generate test data:

public static class Helper
    {
        public static IEnumerable<object> Get5Employees()
        {
            var employees = GenerateData.GetEmployees(5);
            return (IEnumerable<object>) employees;
        }

        public static IEnumerable<object> Get20Employees()
        {
            var employees = GenerateData.GetEmployees(20);
            return (IEnumerable<object>) employees;
        }
        public static IEnumerable<object> Get5Cars()
        {
            var cars = GenerateData.GetCars(5);
            return (IEnumerable<object>) cars;
        }

        public static IEnumerable<object> Get13Cars()
        {
            var cars = GenerateData.GetCars(13);
            return (IEnumerable<object>)cars;
        }

        public static IEnumerable<object> Get5EmployeesWithCars()
        {
            var cars = GenerateData.GetCars(5);
            var employees = GenerateData.GetEmployees(5);
            var employeeswithcars = new List<EmployeeWithCar>();
            for (int i = 0; i < cars.Count; i++)
            {
                var employeewithcar = new EmployeeWithCar
                {
                    Car = cars[i],
                    Employee = employees[i]
                };
                employeeswithcars.Add(employeewithcar);
            }
            return (IEnumerable<object>)employeeswithcars;
        }

        public static IEnumerable<object> Get10SequentialInts()
        {
            var intobjects = new int[] {1, 2, 3, 4, 5,6,7,8,9, 10};
            return (IEnumerable<object>) intobjects.Cast<object>();
        }

        public static IEnumerable<object> Get5StringObjects()
        {
            var stringobjects = new string[] {"String1", "String2", "String3", "String4", "String5" };
            return (IEnumerable<object>)stringobjects.Cast<object>();
        }
    }

The Execution results!

image

And the output for each of the result!

image

Enabling Targeted Environment Testing during Continuous Delivery (Release Management) in VSTS

In my previous post “Continuous Delivery using VSO Release Manager with Selenium Automated Tests on Azure Web Apps (PaaS)”, I’ve walked through the steps on how to enable continuous delivery by releasing builds to multiple environments. One caveat that I didn’t focus on is taking the same tests and running it against those target environments.

In this post, I’ll walk you through the steps on enabling or running the same tests targeting those environments that you have included as part of your continuous delivery pipeline. You’re basically taking the same tests however passing in different parameters such as the target URL and/or browser to run.

In a high level, these are the changes. The solution only requires:

· Create runsettings file and add your parameters.

· Modify your code to call the parameters from the .runsettings file

· Changes to the “Test Steps” in VSTS release or build

Create runsettings file

In Visual Studio, you have the ability to generate a “testsettings” file however not a “runsettings” file. What’s the difference between both file formats? Both can be used as a reference for executing tests. However the “runsettings” file is more optimized to be used for running Unit Tests. In this case our tests are unit tests.

For more information see: Configure unit tests by using a .runsettings file

https://msdn.microsoft.com/en-us/library/jj635153.aspx

Essentially:

1) Create an xml file with the extension of: .runsettings (e.g. ContinousDeliveryTests.runsettings)

2) Replace the contents of the file with the following:

<?xml version="1.0" encoding="utf-8"?>

<RunSettings>

   <TestRunParameters>

      <Parameter name="EnvURL" value="http://XXXXX.azurewebsites.net" />

      <Parameter name="Browser" value="headless" />

   </TestRunParameters>

  <!--<RunConfiguration> We will use this later so we can run multiple threads for tests

    <MaxCpuCount>4</MaxCpuCount>

  </RunConfiguration>-->

</RunSettings>

3) Save the file to your source control repository. In my case GIT in VSTS.

Modify your code to call the params from the .runsettings file

In your unit tests where you specify the URL, apply the following changes:

var envUrl = TestContext.Properties["EnvURL"].ToString();

At this point, you can refer the envUrl variable in your code for navigating through your tests.

Changes to the “Test Steps” in VSTS release or build

In VSTS Release Manager, modify the test steps to:

NOTE: As part of your build definition, ensure that you upload your runsettings file as part of your artifacts directory. Here’s a snippet in my build definition step:

CD1

Visual Studio Test step in Release Manager:

For each environment that you deploy your application, modify the Visual Studio Test Step to:

– Specify the runsettings file. You can browse through your build artifacts provided you’ve uploaded your runsettings file as part of your build artifacts

– Override the parameters with the correct value for your environment URL

image

One you make the following changes, your tests will execute targeting those environments specified in your continuous delivery pipeline.

Continuous Testing in VSO using Selenium WebDriver and VSO Test Agents (On-Premise)

This post walks you through on how to implement continuous testing using the following technologies:

  • Using Config Transform to configure your tests to run multiple browsers during runtime
  • Selenium WebDriver – Automation UX framework to drive UX Testing
  • VSO Build – ALM (Application Lifecycle Management) tool suite for storing code (via GIT), creating the build definition (Build) and configuring On-Premise machines as Test Agents

What is the Config transform? This will allow you to change the configuration settings for your app or web configuration files. When you use Selenium WebDriver, you have the option to run your tests using either Chrome, Internet Explorer, FireFox even PhantomJS (Headless testing). See the sample C# code below to generate the correct web driver instance through configuration settings. The method takes in a string value with a default browser of Internet Explorer. Note that sample code below uses try catches to log exception using Log4Net. You can ignore this but just re-use the code for creating the proper Selenium WebDriver

NOTE: If you’re not familiar on using Selenium to run automated UX testing, see these samples below:http://docs.seleniumhq.org/docs/03_webdriver.jsp

public static T OpenBrowser<T>(string browser = "iexplore", bool useExistingBrowser = false, bool usebrowserstack=false)
  {
            string error = string.Empty;

            try
            {
                if (!string.IsNullOrEmpty(browser))
                {
                    browser = browser.ToLower();
                    switch (browser)
                    {
                        case "firefox":
                            IsFirefox = true;
                            return (T)Convert.ChangeType(ReturnDriver<FirefoxDriver, FirefoxProfile>(useExistingBrowser, ref _firFoxDriver,
                                Firefoxprofile), typeof(FirefoxDriver));
                        case "chrome":
                            IsChrome = true;
                            return (T)Convert.ChangeType(ReturnDriver<ChromeDriver, ChromeOptions>(useExistingBrowser, ref _chromeDriver,
                                Chromeprofile), typeof(ChromeDriver));
                        case "headless":
                            return (T)Convert.ChangeType(ReturnDriver<PhantomJSDriver, PhantomJSOptions>(useExistingBrowser, ref _phantomjsdriver,
                                phantomjsoptions), typeof(PhantomJSDriver));
                    }
                }

                // ie (internet explorer)
                IEprofile.IgnoreZoomLevel = true;
                IEprofile.EnsureCleanSession = true;
                IsIE = true;
                return (T)Convert.ChangeType(ReturnDriver<InternetExplorerDriver, InternetExplorerOptions>(useExistingBrowser, ref _internetDriver, IEprofile), typeof(InternetExplorerDriver));              
            }
            catch (Exception ex)
            {
                if (string.IsNullOrWhiteSpace(ex.Message))
                    _Testlog.Info(MethodBase.GetCurrentMethod().Name + " Results = Success");
                else
                    _Testlog.Info(MethodBase.GetCurrentMethod().Name + " Results = " + ex.Message);
                throw new ArgumentException(ex.Message, ex.InnerException);
            }
            return default(T);
  }

The Generic Type to return the driver is:

NOTE: I’ve hardcoded the value of T as IWebDriver instance to maximize the browser. You don’t have to do this but since we’re using Selenium WebDriver, I’ll just embed it on this method. You can also change the T type to any UX automation.

private static T ReturnDriver<T, TT>(bool existing, ref T driver, TT profile)
        {
            string error = string.Empty;
            
            try
            {
                if (driver == null || existing == false)
                {
                    driver = (T)Activator.CreateInstance(typeof(T), profile);
                }
                ((IWebDriver)driver).Manage().Window.Maximize();
            }
            catch (Exception ex)
            {
                if (string.IsNullOrWhiteSpace(ex.Message))
                    _Testlog.Info(MethodBase.GetCurrentMethod().Name + " Results = Success");
                else
                    _Testlog.Info(MethodBase.GetCurrentMethod().Name + " Results = " + ex.Message);
                throw new ArgumentException(ex.Message, ex.InnerException);
            }
            return driver;
        }

This method takes in a string param. You can pass the value from the config file and if you use config transforms, you can change the run behavior of the browser just by passing in the correct app config value.

Almost forgot! Download the config transform here:  Configuration Transform
https://visualstudiogallery.msdn.microsoft.com/579d3a78-3bdd-497c-bc21-aa6e6abbc859

Here’s a look at what the config transform would look like in your project:

clip_image002

The particular key that I used and configured would look very similar to this (this is from, App.Chrome.config):

<!-- valid options for Browsers are: chrome , firefox, iexplore, headless -->
    <add key="Browser" value="chrome"  xdt:Transform="Replace" xdt:Locator="Match(key)" />

Source Control is GIT in VSO

Given that we’re using VSO as our ALM suite, we’ve opted to use GIT as our backend source control system. This make it’s easier for configuring your build definition since all source code is stored in VSO

Configuring On-Premise VSO Test Agents

The next step is to ensure that you have On-Premise Test Agents that VSO can talk to and execute the tests. For this follow the steps on this article to configure your VSO Test Agents. Note that in VSO, build and test agents are now in the same machine. Also, note that this article talk about On-Premise TFS HOWEVER, the same applies in VSO. You have to go to your VSO instance (https://xxxx.visualstudio.com) and configure your agent pools. The rest is shown belowhttps://msdn.microsoft.com/Library/vs/alm/Build/agents/windows

Let’s do a quick check!

  • Automated UX Tests have been developed in and uses configuration settings to drive the browser driver – Check!
  • Installed Configuration Transform and configured my test project with all appropriate config settings – Check!
  • Test Code stored in GIT (VSO) – Check!
  • On-Premise VSO Test Agents Configured – Check!

Once all of these have been verified then the final step would be stitching and putting all of these together via VSO Build.

Configuring VSO Build for Continuous Testing:

Step 1: Configure Build Variables:

clip_image004

 

 

Step 2: Create Visual Studio Build step:

First step is to build the test solution/project. This is pretty straightforward. On the Solution textbox, browse through the .sln file that you’ve checked in source control (in this case GIT)

clip_image006

Step 3: Deploy the Test Agent on the On-Premise Test Machines:

NOTE: Before completing this step, ensure that you’ve properly configured your test machines. Follow the article below. This articles walks you through creating machines groups for your team project:

http://blogs.msdn.com/b/charles_sterling/archive/2015/07/05/integrating-testing-into-the-ci-and-cd-pipelines.aspx

The key here is ensuring that:

· You have 1 test machine group that has all the test agents configured correctly

· The $(TestUserName) must have admin privileges on the test agents

clip_image008

Step 4: Copy The Compiled Test Assemblies To The On-Premise Test Agents:

The key to this step is ensure that you copy the compiled test assemblies to the right location. $(Build.Repository.LocalPath) is the directory from the build server where “Destination Folder” will copy the test assemblies to the target test agent machine.

clip_image010

Step 5: Execute the Tests:

Nothing special here. Just make sure that you’re reference the correct Test Drop Location. Just copy the Destination Folder from the previous step:

clip_image012

If you configured it correctly, you should get a successful build! Now the result of the build depends on the Pass/Fail results of the tests. In this case, I’ve intentionally Failed 1 automated test to trigger a build failure that coincides to a failing test. Passing the fail test in the future will result to a complete build

clip_image014

clip_image016

In your VSO Home or Team (Dashboard) Pages, you can pin the build trending charts to see PASS/FAIL Patterns for your UX Automation

clip_image017