DevOps Pipeline (1), Management

Since Microsoft introduced Azure in 2010, we have changed our computing habits a lot, basically from local or promises to cloud. This is a huge change and a huge leap. Looking back, we did have a lot of experience indirectly or directly working on or related to Azure, or associated DevOps.

In 2011, I participated in a project in Health Care field, a huge project with 10 terabytes of data in production and 5000 database transactions per second at peak time. We have 200 developers involved in this project, divided in 20 feature Agile Feature teams. We use TFS server, auto built the app when anyone checked in any changes to the server. At that time, it might not be Azure DevOps, but the operation was somewhat similar with the concept of CI (Continue Integration) and CD (Continue Deployment).
I had worked on a project, the client wanted to move all current Apps from TFS promises Server into DevOps Cloud associated with Git as source control management. We worked with MS Pipeline intensively.
Another project, with Fortify Scanning, a security scanning, we need to set up an Azure Pipeline for the automation process with Fortify as core functionality.
In recent years, more and more companies moved or moving to Cloud, such as Azure, associated with DevOps, Pipeline.

All of them require us, even as a developer, to be familiar with Azure, and DevOps. I had article series about Source Control (Repos), this series is about Pipelines. The style of the article series is similar to the Repos series, These are not system summary, or introduction, or learning notes, these are just gathering what I have experienced and I assume I understood and shared with myself and you. The following are the topics I have now and will complete within days or weeks. Later on, I will add some more materials according to new experience or summary.

A - Introduction

This article topic is import/export and clone pipelines. The following is the content of this article

A - Introduction
B - Get Pipeline Manager through GUI
C - Get Pipeline Management through YAML
D - What is YAML
E - Comparison between GUI and YAML
F - A Sample
G - Summary

If we had any experience beyond just using pipeline, for example, we need to setup, manage or change some feature of an existing pipeline, then we need to know the way to handle pipeline. There are, in fact, two ways to do so, one is through GUI, another one is through a script language, YAML. We will discuss this issue in this article.

B - Get Pipeline Manager through GUI

In a pipeline, we choose a specific one:

Click Edit:

we get the pipeline GUI, edit panel:

C - Get Pipeline Management through YAML

For a specific pipeline, Export YAML file:

we got the file downloaded locally:

Open the file in an editor, such as VS Code or Visual Studio

# Variable 'Build32bitPlatform' was defined in the Variables tab
# Variable 'Build64bitPlatform' was defined in the Variables tab
# Variable 'BuildAllPlatforms' was defined in the Variables tab
# Variable 'BuildDefinitionName' was defined in the Variables tab
# Variable 'DaysDelay' was defined in the Variables tab
# Variable 'DebugPref' was defined in the Variables tab
# Variable 'DisableInteliTrace' was defined in the Variables tab
# Variable 'forceClean' was defined in the Variables tab
# Variable 'InstallerPath' was defined in the Variables tab
# Variable 'MSDeployPackage' was defined in the Variables tab
# Variable 'MSDeployPackagex' was defined in the Variables tab
# Variable 'ProductHotFixVersion' was defined in the Variables tab
# Variable 'PSVerbose' was defined in the Variables tab
# Variable 'TestRoot' was defined in the Variables tab
# Variable Group 'VS2022 Specific' was defined in the Variables tab
# Variable Group 'CAE6.0_CurrentBuild' was defined in the Variables tab
trigger:
  branches:
    include:
    - refs/heads/master
  paths:
    exclude:
    - Cap
  batch: True
name: $(BuildDefinitionName)_$(date:yyyyMMdd)$(rev:.r)
jobs:
- job: Phase_1
  displayName: Build
  timeoutInMinutes: 30
  cancelTimeoutInMinutes: 1
  pool:
    name: CAEBUILDS
    demands:
    - Build60CI
  steps:
  - checkout: self
    clean: $(ForceClean)
  - task: CmdLine@2
    displayName: copy  assembly version.cs
    inputs:
      script: "REM this saves changing all the csproj files in test automation\nxcopy  /y \"$(SourceRoot)\\Source\\Common\\Include\\AssemblyVersion.cs\"  \"$(SourceRoot)\\TestAutomation\\CAE\\Source\\Common\\Include\\*.*\" \n\n\n"
  - task: MSBuild@1
    name: MSBuild_12
    displayName: Build Installer Tests
    inputs:
      solution: $(sourceRoot)/Automation/Trizetto.CareAdvance.Automation.Installers/Trizetto.CareAdvance.Automation.Installers.sln
      msbuildArchitecture: x64
      msbuildLocation: $(MSBuildPath)
      platform: $(BuildAllPlatforms)
      configuration: $(BuildConfiguration)
      msbuildArguments: /restore $(DisableInteliTrace)
      logProjectEvents: true
  - task: MSBuild@1
    name: MSBuild_14
    displayName: Build Legacy Automation Tests
    inputs:
      solution: $(sourceRoot)/Automation/AutomationTests.sln
      msbuildArchitecture: x64
      msbuildLocation: $(MSBuildPath)
      platform: $(BuildAllPlatforms)
      configuration: $(BuildConfiguration)
      msbuildArguments: /restore $(DisableInteliTrace)
      logProjectEvents: true
  - task: MSBuild@1
    displayName: Build New Environment Tests
    inputs:
      solution: $(sourceRoot)/Automation/EnvironmentTests.sln
      msbuildArchitecture: x64
      msbuildLocation: $(MSBuildPath)
      platform: $(BuildAllPlatforms)
      configuration: $(BuildConfiguration)
      msbuildArguments: /restore $(DisableInteliTrace)
      logProjectEvents: true
  - task: MSBuild@1
    displayName: Build Data Integration Tests
    enabled: False
    inputs:
      solution: $(sourceRoot)/Tests/DataIntegration/DI.Tests.sln
      msbuildLocationMethod: location
      msbuildVersion: 14.0
      msbuildLocation: $(MSBuildPath)
      platform: $(BuildAllPlatforms)
      configuration: $(BuildConfiguration)
      msbuildArguments: /restore $(DisableInteliTrace)
      logProjectEvents: true
  - task: MSBuild@1
    displayName: Build Workflow Test Automation
    enabled: False
    inputs:
      solution: $(SourceRoot)/AutomationManager/AutomationManagerAll.sln
      msbuildLocationMethod: location
      msbuildVersion: 14.0
      msbuildLocation: $(msbuildpath)
      platform: $(BuildAllPlatforms)
      configuration: $(BuildConfiguration)
      msbuildArguments: /restore /p:EnableIntelliTrace=false
      clean: true
      logProjectEvents: true
  - task: PowerShell@1
    displayName: Create Workflow Classes Output Folder
    continueOnError: True
    enabled: False
    inputs:
      scriptType: inlineScript
      inlineScript: New-Item -ItemType Directory -Force -Path $(SourceRoot)\AutomationManager\bin\Framework\WorkflowClasses
  - task: CmdLine@1
    displayName: Build Workflow Classes
    continueOnError: True
    enabled: False
    inputs:
      filename: '$(SourceRoot)\AutomationManager\bin\Framework\AutomationManagerTool.exe '
      arguments: '"$(SourceRoot)\AutomationManager\bin\Framework\AutomationWorkflows" "$(SourceRoot)\AutomationManager\bin\Framework\WorkflowClasses"'
      workingFolder: $(TestRoot)\AutomationManager\bin\Framework
  - task: CmdLine@1
    displayName: Build Dynamic Test Library
    continueOnError: True
    enabled: False
    inputs:
      filename: '"$(msbuildpath)\Roslyn\csc.exe"'
      arguments: /t:library /out:$(SourceRoot)\AutomationManager\bin\Framework\DynamicTest.dll $(SourceRoot)\AutomationManager\bin\Framework\WorkflowClasses\*.cs  /r:$(SourceRoot)\AutomationManager\bin\Framework\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll,$(SourceRoot)\AutomationManager\bin\Framework\AutomationCore.dll
  - task: MSBuild@1
    displayName: Build Workflow Test Automation
    inputs:
      solution: $(testroot)/AutomationManager/AutomationManagerAll.sln
      msbuildLocation: $(msbuildpath)
      platform: $(BuildAllPlatforms)
      configuration: $(BuildConfiguration)
      msbuildArguments: /restore /p:EnableIntelliTrace=false
      clean: true
      logProjectEvents: true
  - task: PowerShell@1
    displayName: Create Workflow Classes Output Folder
    continueOnError: True
    inputs:
      scriptType: inlineScript
      inlineScript: New-Item -ItemType Directory -Force -Path $(testroot)\AutomationManager\bin\Framework\WorkflowClasses
  - task: CmdLine@1
    displayName: Build Workflow Classes
    continueOnError: True
    inputs:
      filename: '$(testroot)\AutomationManager\bin\Framework\AutomationManagerTool.exe '
      arguments: '"$(testroot)\AutomationManager\bin\Framework\AutomationWorkflows" "$(testroot)\AutomationManager\bin\Framework\WorkflowClasses"'
      workingFolder: $(TestRoot)\AutomationManager\bin\Framework
  - task: CmdLine@1
    displayName: Build Dynamic Test Library
    continueOnError: True
    inputs:
      filename: $(AMcscCommand)
      arguments: /t:library /out:$(testroot)\AutomationManager\bin\Framework\DynamicTest.dll $(testroot)\AutomationManager\bin\Framework\WorkflowClasses\*.cs  /r:$(testroot)\AutomationManager\bin\Framework\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll,$(testroot)\AutomationManager\bin\Framework\AutomationCore.dll
...

D - What is YAML

YAML is a human-readable data-serialization language. It is commonly used for configuration files and in applications where data is being stored or transmitted. YAML targets many of the same communications applications as Extensible Markup Language (XML) but has a minimal syntax which intentionally differs from SGML --- the above definition is from wiki.

When I first time touch the word, YAML, I thought that must be another Extensible Markup Language, such as XAML used in WPF, there could be another one called ZAML, and so on. Actually, my guess logically was correct, but in fact, that was wrong.

Let us briefly have an overview of the histories of both XAML and YAML:

XAML, from wiki, (Extensible Application Markup Language) is a declarative XML-based language that Microsoft developed for initializing structured values and objects. It was introduced in 2008, associated with Avalon being the code-name for Windows Presentation Foundation (WPF).

It is used extensively in Windows Presentation Foundation (WPF), Silverlight, Workflow Foundation (WF), Windows UI Library (WinUI) and Universal Windows Platform (UWP).

YAML was first proposed by Clark Evans in 2001,[13] who designed it together with Ingy döt Net[14] and Oren Ben-Kiki.[14] Originally YAML was said to mean Yet Another Markup Language,[15] because it was released in an era that saw a proliferation of markup languages for presentation and connectivity (HTML, XML, SGML, etc). Its initial name was intended as a tongue-in-cheek reference[16] to the technology landscape, referencing its purpose as a markup language with yet another construct, but it was then repurposed as YAML Ain't Markup Language, a recursive acronym, to distinguish its purpose as data-oriented, rather than document markup.

Now, we know: YAML --- YAML Ain't Markup Language, It is better for us just to view YAML as a scripting language to run pipeline.

E - Comparison between Pipeline GUI manager and YAML file

In my limited experience working with pipelines, unless you are a very junior role, everything is dependent on system people or other senior developers, you sooner or later need to handle, either manager or setup, pipeline yourself, so you'd better know something about pipelines. My experience to touch pipeline even as a developer:

In some cases, when you add an app into pipeline, you may need to write a pipeline, or at least choose a pipeline from a model pool to add it into your app, and check into DevOps to make the auto compiling;
In some cases, you might need to change a parameter, such as .exe name, to the pipeline to make it run well;
In some cases, you might need to create, or edit, a pipeline for a new project for build, deployment, or security checking, such as Fortify, SonaType, WebInspect scanning.

Let us start from the essential, comparing the pipeline GUI manager and YAML file to get What they are:

Variables:

From the YAML file, the first 16 lines are variables

Defined in Variables Tab in GUI, Click Variables:

We get the Variables Page:

This page is equivalent to YAML file for the first 16 lines.

Trigger:

We collapse the YAML file, we get two parts of the YAML:

Trigger
Job

Uncollapse the Trigger:

this is equivalent to the GUI of Trigger:

Comparing them, if we uncheck Enable continuous integration in GUI, we got nothing:

when it checked, we can see the pair is identical (see matched color):

YAML:

GUI:

Job::

uncollapse job --- we have one job here:

Uncollapse Job:

It must match the Build:

Like this:

Tasks:

We have 13 tasks from YAML:

In GUI, we can see 5 tasks + 2 groups:

Right Click one group => Manager Task Group

Open the group, we have 4 tasks in it:

Another one is the same, so we have 13 tasks in GUI too.

We compare one pair:

YAML:

GUI:

F - A Sample:

We got an error message from pipeline running:

Check from YAML file for the keyword to cause the issue, we found the location:

Then we located it from the GUI easily:

G - Summary:

This article discusses MS Azure DevOps Pipelie's management tool: GUI and YAML, and compares them in order to further discussion about how to use them.

Reference