The road to


(A CICD post-mortem)

  • Dad of a 8 year old

  • Senior Developer @ Ops Management Team

  • ~10 years exp as a developer

Terrance Smith


  • Where we started (the setup)
  • The Road (the conflict)
  • Where we stand (the payoff?)

Where we started

Our Back-end

  • Asp.Net Web API 2 (Service)
  • DynamoDb (DB)
  • Psake (Task Runner)
  • Gis.StratusBuild


Our Front-end

  • Angular 4*
  • Angular Cli
  • node\npm 

What could possibly go wrong?

the road

First we met jenkins

Road Bumps

  • Lack of Jenkins Knowledge
  • Lack of  groovy knowledge
  • VSTS Authentication Issues
  • Permissions\Access Issues for Remote Team (ongoing)

jenkins Lessons learned

  • Setup email notifications ASAP for remote folks
  • Navigation of the UI is Essential
  • Be picky about plugins
  • Test run locally where possible (setup a Jenkins server on your box)

jenkins Lessons learned 2

  • Filter your branches as much as possible for improved performance
  • Use the "Replay" feature in the Build to test changes to your jenkinsfile
  • Because you will live in the Console Output, you should log judiciously.

then we paid the aws toll

Cost of Crossing

  • Cloud Formation Template knowledge
  • Overwhelming number of tools
  • Acceptance that you will not remote into every box you work on

Aws Lessons learned

  • Reminder: You should treat your servers as cattle, not as pets.
  • Be prepared to spend a log of time tweaking Cloud Formation Templates
  • Leverage CodeDeploy and  CloudFormation Logs where possible to troubleshoot deployment issues 

Aws Lessons learned 2

  • If your data requires complex filtering\conditionals then consider a traditional db
  • If your data is lightweight and simply structured and required little configuration then consider DynamoDB
  • Use the AWS Cli for all configuration tokens\keys etc...

Then we hit .net build ROADBLOCKS

Road Blocks

  • Incorrect versions of msbuild
  • Invalid\Missing Web Targets
  • Binding Redirect Errors
  • Environment Inconsistencies 
  • Swagger Configuration Errors
  • Mixing Net Standard 1.4-2.0 code with Full Framework

.net build Lessons learned

  • Do not update nuget assemblies carelessly
  • Do not mix .net standard versions
  • YAGNI on refs; Remove what you don't need

.net build Lessons learned

  • Verify your build version is the same everywhere
  • Consider using Paket to lock down build versions
  • We are still trying to figure this one out.... :(

finally we went offroad front-end 

Off Road Action

  • Scripting integrations
  • IIS Configuration for static js 
  • npm version issues
  • Ng Tool versions
  • npm Global conflicts
  • Sass compiler issues

Front-end Lessons learned

  • Embrace angular cli tooling
  • Rely on local node_modules where possible
  • Avoid installing global npm packages on build server


Front-end Lessons learned 2

  • Plan for non-trivial effort when upgrading versions of angular and cli
  • Consider using Angular Cli Environment variables for use with deployment
  • Consider Static Hosting on IIS vs Serving via
  • Build configuration management tooling into application for fewer endpoint config headaches

Where we stand


The Road to DevOps

By Terrance Smith

The Road to DevOps

  • 1,460