Similar presentations:
Build and Gated Check-in
1. Build and Gated Check-in
2. Goals
Develop understanding of:• Build system architecture
• Differences between AX6.x, and AX7
• Current system state
• Roadmap
• Common operations scenarios
3. Knowledge
• Process FAQ - http://fargobuild/processfaq/• AX FAQ - http://fargobuild/axfaq/
• SharePoint sites:
ESS Ops
Build Infrastructure
Dev Infrastructure
ESS Documents
• Happydev - http://happydev
4. Overview
Source ControlBB
Execution
Build Reporting – DBs / Services
Checkin
Wizard
BB
Wizard
Code
Review
Build
Tracker
Queue
Processor
BuildTracker
Checkin Infrastructure
Gate execution
Official build Execution
Eng.
Fundamentals
Developer makes a change
SCRAM
SCRAM
WTT
WTT
Lab Services
CodeFlow
Enlistme
CoreXT
PRSS
QE, SCA,
SymIndex
BGIT
5. Build Central Infrastructure
6. Lab Footprint
DCSRDCHKINFSABuild Drops
BuildTracker
Gates
Code review pack
share
Dcsrdchkinfs01
RR1SQLB02
dcsrdbldfsa
Dyn
Log/Pack share
RDVMHL1109-05
Hosts Dyn DFSN
dcsrdblddrop03
RR1SQLB03
DCSRDBTRACKER01
Queue Processor
Synthetic checkin
service
DCSRDGTDSYMA
Dcsrdgtdfs01
BuildService
Log Shares
Reporting parsers
Tools share
Pending pack share
Machine reporting
Status emails
Schedules
maintenance tasks
Symbol file share
DCSFRFargoBuild
Hosts FAQ sites
DCSRDRPTSQLA
DynamicsAXBuilds
FAQData
ProcessFAQ
DCSRDBLDIIS3
Hosts Branch Config site
Hosts Autointegrate site
DCSRDBLDSQL01
AutoIntegration
Dcsredbbcfs01
Statistics
UnitTests
7. Source Depot - Overview
Server Info:Hosted/Managed by BGIT
Connect To - vedbuitfsa06:4020
Actual Server - TK5SDSERVDAX01
What to do when you notice the depot
is down
8. Source Depot - Overview Contd.
ProxiesEnhance local user experience
Both BGIT-managed and private
Current proxies : “sd proxies -o”
14 exist today
Administration
• //depot/Admin/…
• SD Admin Explorer - http://toolbox/22929
9. Source Depot - Overview Contd.
Help• Sd help
• Sd help undoc
Protections
• Controls who has permission to what
• Modifying protections
Triggers
• Logic to run on command execution
• Main usage – preventing direct submits
• Modifying triggers
10. Source Depot - Overview Contd.
Maintenance• BGIT-managed updates to server/SD instance
• Remove unused clients
Client components
sd.exe
sdapi.dll
sdb
sdv
Packing tools: sdp, jjpack
11. Source Depot - Branching
MainSource Depot - Branching
BuildServices
Purpose:
Dormant/
Integration
DAX5
DAX5SP1HF
DAX6SE
DAX6HF
DAX6HF_STAB
DAX62Stab
DAX63SE
DAX63HF
DAX63HFSTAB
RainINT
DAX63NS
DAX63NSStab
DAX6_BT
DAX6
DAX62OOBSTAB
DAX62OOB
DAX62
In-Market
DAX62HFSTAB
DAX62HF
LCS
DAX62SE
DAX62CD
AX7
RainMain
Infra
RainMainStab
RainFND
RainStab
RainLCS
RainLCSV5
App<Rel.>
RainFNDStab
RainGL
RainLCSStab
LCS
RainLCSHF
LCSStab
Plat<Rel.>
CoreServices
LCSHF
12. Source Depot - Integrations
Introduction…it is “to propagate changes happened to a namespace (source namespace)
during a certain period of time into another namespace (target namespace)”.
-SD Integrations Guide
Four States to Consider:
1.
2.
3.
4.
State of the source just before a given period of time
State of the source exactly at the end of a given period of time
State of the target at the moment of integration
State of the target at the end of the integration
13. Source Depot - Integrations Contd.
Revision• Versioning at the file level
• Referenced with ‘#’
• Example – sd sync //depot/rainier/rainfnd/foo.txt#2
Changelist
• Collection of file revisions commited at a single point in time
• Referenced with ‘@’, -c <changelist>, <changelist>
• Example – sd describe 123456
Label
• Collection of changelists
• Referenced with ‘@’
• Example – sd sync …@7.0.123.0
14. Source Depot - Integrations Contd.
SD Integrate• Forward Integrate (FI) versus Reverse Integrate (RI)
• Full branch integration
• sd integrate -b <child branch> @<revision>
• Point integration
• Sd integrate -b <child branch> @<rev 1>,<rev 2>
• Manual mapping - renaming or baseless integrates
• sd integrate //depot/b1/foo.txt#1,2 //depot/b2/foo.txt
15. Source Depot - Integrations Contd.
SD ResolveAutomatic resolve – sd resolve -as
Manual resolve – sd resolve
Accept Thiers (at) vs. Accept Yours (ay) vs. manual merge (e)
Conflict/Manual resolution:
>>>> ORIGINAL sources.all#2 ++++++++++++++++++++++++++++++++++++++++++++++++
#endif
MSBUILD_VERSION=4.0
==== THEIRS sources.all#3 ++++++++++++++++++++++++++++++++++++++++++++++++
#endif
==== YOURS sources.all ++++++++++++++++++++++++++++++++++++++++++++++++
#endif
MSBUILD_VERSION=12.0
<<<< END ++++++++++++++++++++++++++++++++++++++++++++++++
16. Source Depot - Integrations Contd.
AutoIntegrate tool• Set of automation wrapping “sd.exe integrate”
• Custom resolvers
• MSBuild custom tasks/targets
• //depot/../Build/AX_Tools/AutoIntegrate/...
• Web Resolve
• Run via command line “Integrate” command or BuildTracker jobs
17. Source Depot - Pack Files
Jjpack.exe• Primary packing format used in build/gates
• Supports integrations
• Does not support ‘undo’ – downgrades file op
SDP.exe
• Does not support integrate options
• Double-click diff
• Used more commonly by Engineers to pass around changes
18. BuildTracker
Our Instance:• http://daxbuild/BuildTracker
BT product site:
• http://buildtracker
BT Lab Solutions DG:
• btracker
19. BuildTracker - Services
RR1SQLB02• Database
RR1SQLB03
• BuildA – Access Management
• BuildK – Manages BT encryption key
• BuildL – Lab Monitoring (WMI monitoring)
• BuildM – Mail and notifications
• Web UI – http://daxbuild/buildtracker
• API - net.tcp://RR1SQLB03:9700/BuildTrackerApi
DCSRDBTRACKER01
• BuildS – Schedules job queues
Client machines
• BuildQ – Worker service
20. BuildTracker - Lab Maintenance
Build Team Services ProjectWindows update on clients
Reports
DB Backups
Auto-archive and drop cleanup
Gate Queue Processor check
BT Automation rules
21. BuildTracker - Lab Maintenance Contd.
Build Tracker Automation Library• Rules framework for enforcing consistent
practices in our BT instance
• Runs daily at 3am PST
• Source in BuildServices at
\build\source\MS.Dynamics.Build\...
22. BuildTracker - BT.exe
• Wrapper for common lab maintenance and reporting functions usingBuildTracker’s APIs
• Available functions differ between BuildServices and product
branches
• Source - \build\ax_tools\buildtracker\bt\...
23. AX7 / Rainier
24. The Product
• CTP Technical Concepts guide on HappyDev25. The Product
• CTP Technical Concepts guide on HappyDev26. Lab Footprint (AX7)
DCSRDCHKINFSACode review pack
share
Build Drops
Test
dcsrdbldfsa
Test Clients
dcsrdblddrop03
Symbol file share
Hosts Dyn DFSN
BuildTracker
DCSRDBTRACKER01
24 machines
2 CPU cores
16GB Memory
Primitive Clients
14 machines
Build Clients
Dyn
Dcsrdchkinfs01
Log/Pack share
104 machines
DCSRDGTDSYMA
Gates
VMs allocated via Freebird
8 CPU Cores
60GB Memory
200 + 500 GB FusionIO-backed
2 CPU Cores
16GB Memory
Stress Clients
7 machines
4 CPU Cores
16GB Memory
Dcsrdgtdfs01
150 machines
VMs allocated via Freebird
8 CPU Cores
48GB Memory
200 + 500 GB FusionIObacked
Allocated 80% primary gate
processing / 20% build leg
RDVMHL1109-05
RR1SQLB02
BuildService
Log Shares
Reporting parsers
Tools share
Pending pack share
Machine reporting
Status emails
Schedules
maintenance tasks
Gate Clients
Queue Processor
Synthetic checkin
service
LKG Service
RR1SQLB03
DCSRDRPTSQLA
DCSFRFargoBuild
Hosts FAQ sites
DynamicsAXBuilds
FAQData
ProcessFAQ
DCSRDBLDIIS3
Hosts Branch Config site
Hosts Autointegrate site
DCSRDBLDSQL01
AutoIntegration
Dcsredbbcfs01
Statistics
UnitTests
27. Ax7 / Rainier Branches
RainINTDAX62STAB -> DAX62 -> DAX6 -> DAX6_BT -> Main
RainMain
BuildServices
RainMainStab
RainFND
RainStab
RainLCS
RainLCSV5
App Servicing
Branches (e.g.
AppCTP8,
AppRTW)
RainFNDStab
RainGL
RainLCSStab
LCS
RainLCSHF
LCSStab
Platform
Servicing
Branches (e.g.
PlatCTP8,
PlatRTW)
LCSHF
CoreServices
28. CoreXT - Introduction
What is it?A layer of rules, processes, features and automation
A common set of tools for developers, testers, builders and labs alike.
Generic and decoupled from the applications being built
Community effort within Microsoft
Existed since 2000 – estimate is used by 1/3 of the company
Latest release – v5.0
More information – http://corext
29. CoreXT - Our Environment
OverviewCurrent version – v3.2
Build.exe to each project node
Dependency Based Build (DBB) enabled
Tracer/Analyzer disabled
No adoption of CoreXT integration targets/properties
CoreXT V5 package manager backported
30. CoreXT - Environment Setup
Entry point• %inetroot%\tools\path1st\myenv.cmd
• Shortcut by:
• Open cmd.exe
• Navigate to enlistment root
• Run “setenv”
Branch-specific configurations
• %inetroot%\build\branchcfg\myenv_<branch>.cmd
31. CoreXT – Package Management
• Branch package dependencies defined in a checked-in packagemanifest
• On CoreXT startup, missing packages are retrieved and installed into a
local cache directory.
• Build processes consume package contents from the local cache
• More information on HappyDev
32. CoreXT - Externals
Public Externals• ANY external reference or toolset
• ANY internal reference or toolset submitted as a binary
• Location – NuGet package repository (previously: %inetroot%\public\ext\...)
Toolsets
• Often accompanied by makefiles/targets which integrate into CoreXT
Approvals
• True external libraries (both Non-Dynamics Microsoft produced or 3rd party)
need to receive LCA approval before check-in.
• External Components Search
33. CoreXT - Build
<Root>Build.exe
• Windows build tool
• Orchestrator for building/traversing
the build graph
• Our version – 6.2.8427.0
• ‘dirs’ and ‘sources’
• amd64 is default build target
Dirs
Dirs
Dependency Based Build (DBB)
• Build node dependencies drive build
graph traversal
• Sources.dep
Sources
Sources
Makefile.def
Makefile.def
MSBuild
WiX
Dirs
Sources
Sources
Makefile.def
Makefile.def
CMD
VC++
34. CoreXT - Build Contd.
PASS[0|1|2]• PASS0 – Generation of files, copying of static content
• PASS1 – Compilation, X++, C#, C++ (both compile and link) all happen here
• PASS2 – Packaging/Binplacing/Setup
• Assembling the content from PASS0 and PASS1 into their final containers
• PASS2_BINPLACE – binplacing of built bits (mostly from PASS1)
Project Inclusion
• Integrating your project into CoreXT’s build
• Dirs, sources, sources.dep, makefile, makefile.inc
35. CoreXT - Build Contd.
Rainier Build TopicsXPPC
Proxies/FormAdaptors
DBSync
Packaging
36. CoreXT - MSBuild
Current Version – v14.0Build.exe to MSBuild Handoff
• \public\ext\tools\msbuild.def inclusion in project makefile
• \tools\path1st\msbuild.cmd wraps msbuild.exe
• MSBUILD_PROJECT in sources
NMake
Build.exe
Makefile.def
Msbuild.def
MSBuild.cmd
MSBuild.exe
My_Project.csproj
37. CoreXT - FxCop
Source:• All Branches: %inetroot%\public\ext\fxcop\...
Runs on:
• Binplace via %inetroot%\tools\build\buildtypes\fxcop_build.cmd
• MSBuild project when RunCodeAnalysis=true
• Nothing currently enforcing correct ruleset here
Relevant Sources Macros:
• Specify additional parameters - FXCOP_USER_FLAGS
• Skip FxCop on Binplace - BINPLACE_FLAGS=--no-postbuild-fxcop
38. CoreXT - FxCop Contd.
Rule Categories• SWI/Stock – rules fulfilling QE requirements or shipped with FxCop
• Example – Require CultureInfo be specified when calling a member that accepts it
• Dynamics source - enforce Org-specific patterns
• Example - Format of Copyright on binary’s properties
• Dynamics Test – enforce patterns on test code
• Example – Enforce inclusion of required test attributes on test cases
39. CoreXT - FxCop Contd.
Common ProblemsMissing reference assembly
Multi-proc/race condition
Deadlock/timeout
Custom rule crashes
Logs
• XML format per scanned assembly
• From Build log root – FXCopLogs%BuildType%
FxCop error codes - http://msdn.microsoft.com/en-us/library/bb429400(v=vs.80).aspx
40. CoreXT - Customization
Stock CoreXT• Most source scripts under \tools\...
• Makefiles under \public\ext\tools\...
• As a rule don’t modify stock CoreXT
Customization points
\build\automation\...
\build\path1st\...
\build\buildtypes\...
\build\sources.all
\build\preenv.cmd
\build\myenv.cmd
41. CoreXT - Automation
CoreXT + BuildTracker• Stock automation scripts - %inetroot%\tools\build\automation\...
• Customized automation scripts - %inetroot%\build\automation\...
bldTree.cmd
bldCommon.cmd
bldDone.cmd
bldStart.cmd
bldShare.cmd
42. CoreXT - Codereview
• Wizard based on AutoProcess console that wraps CodeFlowsubmissions
• Binaries\Configuration - \build\CodeReview\...
• Source - \build\source\checkinwizard\...
• Managed pack share - \\dcsrdchkinfsa\CodeReviews
• Packs retained for ~2 years.
• Command: “Codereview -c <local changelist>”
43. CoreXT - DynCop
• Home-grown static analysis tool• Developed and used initially for CoreXT v3.2 upgrade
• Develop rules which enforce consistency and best practices across the
build environment
• Executes locally, in gates, and LMC
• Source – VSTS DynCop Git repo
• Documentation on HappyDev
• Product teams adopting for enforcing practices in their teams
44. CoreXT – Platform-As-A-Binary
• Historically, AX has been a monolithic codebase• Re-architect product into Platform component and Application component
to be build/deployed/validated/released/serviced independently
• Platform and Application are exchanged with one another as Last Known
Good (LKG) zip packages
• Further work expected to the product and ESS infrastructure supporting
this
• More on LKG Management here
45. CoreXT - Dirs Filtering Framework
• Mechanism used to partition codebase for initialPlatform/Application build separation
• Uses build.exe’s filtering on dirs {tokens}
• Tool source - \build\ax_tools\DirsFilterManager\...
• Executable - \build\ax_tools\bin\DirsFilterManager.exe
• DynCop rules enforcing correct tokens
• HappyDev documentation
46. CoreXT - Dirs Filtering Framework Contd.
• Examples:Build retail –BuildOnly:fnd
Build retail –BuildConfig:Platform
Build retail –ExcludeTokens:app
Source
DIRS= \
AppIL \
Kernel \
AppIL
Kernel
DIRS= \
Foo{!app} \
Bar{app}\
Foo
DIRS= \
Zoo{fnd} \
Bar
Zoo
47. CoreXT - Odds and ends
Aliases• Maps a command to an alias
• Example – ‘codereview’ invokes the script to launch the code review wizard
• Specified globally per branch in %inetroot%\build\aliases\aliases.pub
PrivateDev
• Global private directory
• Include //depot/private/privatedev/developer/<alias>/… in your client spec
• Map to %inetroot%\private\developer\<alias>
• Private aliases
• Add a file \build\aliases\aliases.pub under private directory
48. Gates
49. Gates - Technologies
WTT• Workflow engine
• Machine
management/maintenance
Powershell
• Log parsing
• Reporting
• Queue processor
Managed code
Log parsing
Reporting
Scheduling
Orchestration
Monitoring
Wizards
Web Service
Cmd
• (nearly) All of the above
50. Gates - 50k Feet
Pre Checkin Checks (WTT)Checkin Wizard
User Submits change
Queue Processor
Available machine found.
Checkin Scheduled
Gate Run (WTT)
51. Gates - Submitting
Checkin submit -c123456
Launch Checkin
Wizard
User chooses
changelist files
Checked-in gate
spec XML
Analyze
changelist
User
configures run
(chooses to
run or bypass
optional tasks
User chooses
TFS or PS bugs
Checkin queue
entry
User submits
checkin
End Wizard
Pack file
Checkin Spec XML
52. Gates - “Checkin” Command
Usage: checkin <operation> <operation parameters>Operation
Description
Submit -c <changelist>
Initiate a new checkin. Launch the wizard
Resubmit -id <checkin id>
Resubmit a previous checkin
Abort -id <checkin id>
Abort a current running checkin
Diff -id <checkin id>
Diff the changes for that checkin
Apply -id <checkin id>
Apply the changes from a checkin locally
Repack -id <checkin id> -c <changelist>
Repack a previous checkin’s pack with the contents of the
specified local changelist
Prioritize -id <checkin id>
Prioritize the checkin in the queue
Bypassqueuelock -id <checkin id>
Allow a checkin to run even if the gates are closed for that
branch
Source script: \public\ext\tools\x86\checkin.cmd
53. Gates - Checkin Wizard
• Wizard for submitting to the gates• Based on AutoProcess console framework
• Configuration: \build\CheckinWizard\...
• Source: \build\source\CheckinWizard\...
• Command: checkin submit -c <changelist>
54. Gates - Queue Processor
Core• Orchestrates scheduling of requested check-ins
• Scheduling dimensions:
• Machine availability
• Blocking files
• Branch lockdown
• Main script is written in Powershell
• Cmd wrapper manages logs and invokes core PS script
• Logs retained for 24 hours
• Executes every 5 minutes as a schedule task on gate server
55. Gates - Queue Processor Contd.
Core Contd.• BuildServices branch - %inetroot%\build\wtt_checkin\SetExternalReferences.cmd
• Log location - %QueueProcessLogDirectory%
• Execution server - %QueueProcessServer%
• BuildServices -
%inetroot%\build\wtt_checkin\QueueProcessor\...
BBCheck.exe
• Core scheduling utility to the gate WTT pool
• Wrapped by “checkin” command
• Abort, Submit, Resubmit, Status, Diff, Apply, BypassQueueLock, Prioritize, Repack, debug
• All branches - %inetroot%\build\source\BBCheckSystem\bbcheck\...
56. Gates - Queue Processor Contd.
Merge Resolution Check• Every 15 minutes for branches that have queued, blocked checkins
• 3-way merge attempted on all files involving:
• Depot state of the file
• State of file in checkin blocked on
• State of file in checkin that’s blocked
• 0 conflicts on all blocked files unblocks checkin
• All branches - %inetroot%\build\wtt_checkin\MergeCheck\...
QueueParse.exe
• Part of ReportingParser cycle
• Reflects blocking file state into DB
• BuildServices - %inetroot%\build\source\QueueParse\...
57. Gates - Execution
Checkin is Scheduled and ExecutesMachine allocation/Preparation
(Sync, nuke, uninstall old deployments, apply changelist)
Changelist and checkin
validations
Build
Deploy and Test
Post Steps
(kernel, app, test, debug flavors)
(BVTs, UTs, BP checks)
(Determine Results, Submit CL, Send report email, copy logs,
update bugs, teardown and release machine)
(DynCop,
SDDump
macro
Schedule
Build leg
validators, code coverage
pairing , etc.)
Primitives
Stress
BVT
Return results
Initiate parallel test runs
Schedule Run
Preparation
Pre Checkin
Retail Build (Platform or
Application)
Post-Build Merge
C# UTs
Deploy
X++ CITs
Determine Results
Schedule parallel leg builds
Return results
Build Leg Machine
Debug Build (Platform or
Application)
Determine Results/
End
58. Gates - Execution Contd.
Primary MachineMaintains checkin state
Executes retail build
Schedules leg builds and SCRAM runs
Deploys
Executes unit tests (C# and X++)
Evaluates task results
Submits (or rejects) changelist
Sends alerts
Updates bugs
Hosts checkin build share
• Retained for up to 3 days
• \\<machine name>\CheckinBuilds\<checkin ID>\
59. Gates - Execution Contd.
Legs• Build leg
• Applies change
• Executes a build
• Reports result back to primary machine (via file share)
• Test leg (SCRAM)
• Runs in SCRAM infrastructure
• Consumes build hosted on primary machine
• Returns result summary to primary machine (via SCRAM.exe/XML file)
60. Gates - Execution Contd.
Environment• %binFolder% - %inetroot%\build\wtt_checkin
• %workFolder% - %inetroot%\build\wtt_checkin\work
• %parm<parameter name>% - Value from WTT job parameter
• %workFolder%\WTTFlag.tmp – indicates running in the gates
• %binFolder%\SetExternalReferences.cmd
• Sets things like %StatisticsDatabaseServer%, %QueueProcessorServer%, etc.
61. Gates - Tasks
Task Intro• Single script under %binFolder%
• Invoked from WTT job task
Format
Set up gate-specific variables (includes getting WTT job parameters)
Log task start in [Statistics].[dbo].[GateTask]
Determine if task can run (disabled, pre-req missing, aborted)
Task setup – E.g. restoring DBs, stopping/starting services
Task logic – E.g. running “build” command
Evaluate result – Write as semaphore file to %workFolder%
Finalize the task in [Statistics].[dbo].[GateTask]
Update task detail in [Statistics]
Exit
62. Gates - Tasks Contd.
Primary MachinePreparation
Sync
Apply Changelist
Pre-Checkin Checks
Pre-Build Preparation
Build Retail-platform (if applicable)
Build Retail-application (if applicable)
Post-Build
Schedule Tests
Deploy
MSTest Tests (C# UTs)
Xref-selected tests (if applicable)
Unit tests (X++ CITs)
* Wait
Submit
Bug Update
Log Copy
Dev gate Leg
Preparation
Sync
Apply Changelist
Build Debug-platform
Build Debug-application
Log Copy
SCRAM
• BVT
• Primitive Tests
• Stress Tests
63. Gates - Tasks Contd.
Adding a taskCreate a task script
Create it under %inetroot%\build\wtt_checkin\<task name>.cmd
Add the task to the WTT job (usually requires a new job version)
Update determine results to evaluate if the task ran, passed or failed
Create the task code in [Statistics].[dbo].[GateTaskCode]
Create the task exit codes in [Statistics].[dbo].[GateTaskExitCode]
Update GCM and DB to include configuration of the task.
Create the shell for GateTask to execute it locally (create it under
%inetroot%\build\wtt_checkin\<task name>FE.cmd)
• Update GateTask.cmd to include the option and execution for the task
64. Gates - Fast Build
Gates - Symbol/binary indexing• Every gate build indexed to http://symweb
• Deduplication and filtering of binaries (RemoveDuplicates.exe)
• Location set in branch config as %GateSynShare%
• Process invoked in Post Build
• Build\wtt_checkin\CheckinBuildDrop.cmd
• Space-based retention handled by the file share
• Additional Documentation here
65. Gates - Fast Build Eligibility
Gates - Synthetic Checkins• Service scheduling regular test checkins to assess the stability of a
branch/checkin path
• Windows service hosted on RDVMHL1109-05
• Source in BuildServices branch at \build\source\SyntheticCheckinService
• Current flavors:
• SyntheticCheckin – default type, executes like a typical checkin in the branch
• BuildOnly – executes a checkin that just does a build in a dedicated pool
• Differentiated in reporting on [Statistics].[Stats].[SyntheticType]
66. Gates - Fast Build Execution
Gates - Reporting / Alerts• MBSUSP
• MBSRPT
Status
Volume
Build Breaks
Offline Machines
Queue Cycle Report
…many many more…
67. Gates - Fast Build : FastApp
Gates - Statistics DB / BuildServiceStatistics
Primary repository of gate execution and configuration data
Lives on dcsredbbcfs01
FAQ on common tables
Exposed via BuildService (for new development)
BuildService
WCF Service
Expose gate data from [statistics], [unittests]; build data from [dynamicsaxbuilds]
Source in BuildServices branch - \build\source\BuildService
Hosted on dcsrdgtdfs01
68. Gates - Fast Build : NightlyBootstrap
Gates - DevelopmentTask Source
• %inetroot%\build\wtt_checkin\...
Tools Source
• %inetroot%\build\ax_tools\...
• %inetroot%\build\source\...
• %inetroot%\build\scripts\...
WTT Artifacts
• Checkin jobs - $\GatedCheckin
• Utilities/Maintenance - $\GatedCheckin\Utilities
69. Gates - Symbol/binary indexing
Gates - Development Contd.Maintenance and core tools
• BuildServices branch
• %inetroot%\build\wtt_checkin\MachineConfiguration\...
• %inetroot%\build\wtt_checkin\
Test environment
• Run locally using “gatetask” command
• Redirect to private WTT pools
• Create new synthetic checkin type
70. Gates - Synthetic Checkins
Gates - Monitoring / AnalysisGated Checkin Monitor (GCM)
• Gate/build monitoring dashboard
• Quick access to common actions (e.g. Resubmit, diff, incident creation)
• Gate configuration
• Queue processor by branch
• Gate tasks by branch
• Source in ‘BuildServices’ branch
• %inetroot%\build\source\GatedCheckinMonitor\...
Logs
• Console log vs. build.exe vs. detail log
71. Gates - Reporting / Alerts
BuildTracker Managed Builds72. Gates - Statistics DB / BuildService
BuildTracker Builds - Official BuildsPurpose
• Produce a ship-quality build
• All infrastructure in place to be ‘All On’
• Provide a rollup of changes to validate against
Cadence
• Runs daily with sporadic OOB runs
• Kick-off staggered across branches
• RainMain* – 9pm PST
• RainFND* – 10pm PST
73. Gates - Development
BuildTracker Builds - Buddy BuildsPurpose
• Provide engineers with a private build containing their pending changes to be used in
pre-checkin validation.
Cadence
• On-demand/user initiated
• Initiated either directly from the BuildTracker web UI or by running “buddybuild –c
<changelist>” in CoreXT
• Permissions controlled by groups:
• redmond\bbscramu
• europe\axsrcacc_ram
Resources
• Jobs– SCRAM Buddy Build-Redmond – Platform, SCRAM Buddy Build-Redmond Application
• Machine pool – Redmond – Rainier Buddy Build
• Drop share - \\dcsrdblddrop03\Build1\BuddyBuilds\Rainier
• Retention is space-based and managed external to BuildTracker
74. Gates - Development Contd.
BuildTracker Builds - Buddy Build Wizard• Binaries/Configuration - %inetroot%\build\BBCheckinWizard
• Source locations
• Source control grid (first screen):
build\Source\CheckinWizard\src\CheckinComponents\Generic\
• Buildtracker screen (second screen):
build\BTCheckinWizard\ExternalContributors\Produc\
• Managed pack share
• Location - \\dcsrdblddrop03\BuddyBuildPackages
• Retention – 30 day policy
75. Gates - Monitoring / Analysis
BuildTracker Builds - Rolling BuildsPurpose
• Provide faster build/validation cycle than nightly official builds
Cadence
• RainFND Rolling Master
• 4x per day
• Initiates PDL runs per build
76. BuildTracker Managed Builds
BuildTracker Builds - JobsMain Build
Buddybuild
Localization (Static)
Retail Build
(Platform or
Application)
Rolling Platform Build
Localization (Static)
Debug Build
(Platform or
Application)
Retail Build
(Platform or
Application)
Localization (Static)
Debug Build
(Platform or
Application
Retail-Platform Build
AxPackage_Contoso
AxPackage_Contoso
Code Sign (CD and
Services)
BVT
TSImport
AxPackage_Contoso
Generate BacPac
Generate BacPac
Translation-submit
77. BuildTracker Builds - Official Builds
BuildTracker Builds – Master BuildsMaster
Main Build
Post Build Tests
LMC
DevFabric BATs
X++ Tests - Full
Stress Tests
Managed flavors
APIScan
FxCop
DynCop
ModernCop
C# Unit Tests
Primitive Tests
BVT Stabilization
LKG Compatibility
BinScope
CodeSign
Check
BinSearch
PoliCheck
Rolling Master
Rolling Platform Build
Post Build Tests
PDL
Primitive Tests
Stress Tests
• Collection of child jobs to execute as a single job
instance
• Typically include a build job and some post build
activities to run if the build was successful
• Usually what daily schedules are set to kick off
78. BuildTracker Builds - Buddy Builds
CodeSign• Authenticode versus Strong Name
• Signing process
• Delay signing
• Authsign.exe
• Certificates
• Non-standard formats/processes
CSPKG
APPX
ClickOnce/VSTO
VSIX
Packages (general)
Links:
• CodeSign – http://codesign.gtm.microsoft.com/
• CodeSignInfo – http://codesigninfo/
79. BuildTracker Builds - Buddy Build Wizard
LocalizationLSBuild
• Command line build tool to create localized product files
• Uses English version of file as input
• Consumes LCL files provided by MBSI
• %inetroot%\source\translation\LCL
• Use various parsers to determine what to do
• Only one custom one – KTD files
80. BuildTracker Builds - Rolling Builds
Localization - Contd.PseudoLoc
• Simulated localized content
• Intended to catch localization problems as early as possible
• Pseudoloc examples from AX 2012 R3
Build Legs
• Localization-initialize – copies drop\translation to release share
• Localization-server – Builds localized KTD files
• Retail-Platform - \source\Platform\Localization\GeneratedPASS2
Retail-Application - \source\net64resources
• Build localized resource files
• %inetroot%\build\localization\localization.binaries.config.xml
• Translation-submit – Signs and submits localized files to \source\translation\
81. BuildTracker Builds - Jobs
QE / LMCQuality Essentials (QE)
• Set of validations focused on ensuring overall release quality
Regulatory/legal
Industry standards
Customer Experience
Security
Consistency with Microsoft Engineering Policies
• More information - http://qe/
Last Minute Checks (LMC)
• Executes QE and Org-specific build quality checks
• Example from DAX62
82. BuildTracker Builds – Master Builds
Post-ship activities• Local and long-term backup
• Source code archive
• Symbol index/archive
83. CodeSign
Visual Studio Online (VSO)• What is VSO?
• Our instance: http://msdyneng.visualstudio.com
• VSO-hosted
Source control via Git
Work item tracking
Build definitions
Build requests/scheduling/reporting
• On-Premises
• Build controller/agent – RBLD0071AC-TFS
• Build drops - \\DCSRDBLDDROP04\Build1\Drop
84. Localization
In-Market85. Localization - Contd.
AX6.x Product• AX 2012 Technet site
• AX 2012 Architecture
• AX 2012 MSDN Developer site
86. QE / LMC
AX6.x Product Contd.AX32.exe – Win32 client
AX32Serv.exe - Server component running as a Windows Service
MorphX – development environment
Model store – database storing the P-code representing the application
inside the platform
• AXDBBuild_Model – build-time instance name
• AXDBDev_Model – runtime instance name build/gates/deploy.cmd
• AXDB – common runtime instance name on other lab machines
• Business database – application database housing the data model for
business logic
• AXDBDev – runtime instance name
87. Post-ship activities
Lab Footprint (In-Market)Build Drops
Gates
dcsrdblddrop02
dcsrdbldtempfs1
dcsrdblddrop03
dcsrdbldtempfs2
dcsrdbldfsa
BuildTracker
RR1SQLB02
RR1SQLB03
Dcsrdchkinfs01
Dcsrdgtdfs01
Build Clients
~50 machines
Log/Pack share
VMs allocated via Freebird and
UIP-managed physical machines
2-16 CPU Cores
8-32GB Memory
250GB – 2TB enlistment drives
BuildService
Log Shares
Reporting parsers
Tools share
Pending pack share
Machine reporting
Status emails
Schedules
maintenance tasks
Gate Clients
110 machines
VMs allocated via Freebird
and UPI-managed physical
machines
2-16 CPU Cores
8-32GB Memory
250GB – 2TB enlistment
drives
Allocated 80% primary gate
processing / 20% build/test
leg
RDVMHL1109-05
Queue Processor
Synthetic checkin
service
DCSRDCHKINFSA
Code review pack
share
DCSFRFargoBuild
Hosts FAQ sites
Dyn
Hosts Dyn DFSN
DCSRDRPTSQLA
DynamicsAXBuilds
FAQData
ProcessFAQ
DCSRDBLDIIS3
Hosts Branch Config site
Hosts Autointegrate site
DCSRDBLDSQL01
AutoIntegration
Dcsredbbcfs01
Statistics
UnitTests
88. Visual Studio Online (VSO)
In-Market BranchesMain
BuildServices
DAX5SP1HF
DAX6SE
DAX6HF
DAX6HF_STAB
DAX63SE
DAX63HF
DAX63HFSTAB
DAX63NS
DAX63NSStab
DAX6_BT
DAX6
DAX62OOBSTAB
DAX5
DAX62OOB
DAX62
DAX62HFSTAB
DAX62HF
DAX62SE
DAX62Stab
DAX62CD
89. In-Market
Servicing Concepts• Dependency toolset
• Hotfix
• Binary vs. Application
• Rollup
• Cumulative Update (CU)
90. AX6.x Product
CoreXT - Our EnvironmentOverview
• Current version – v1
• Build.exe to each project node
• SYNCHRONIZE_BLOCK, SYNCHRONIZE_DRAIN and dirs file ordering used to
drive build ordering
• Custom ‘mydirs’ logic used to partition the build tree corresponding to
custom build types
• Binplace, Placefiles/miscfiles both supported
91. AX6.x Product Contd.
CoreXT - MSBuild• Current Version – v4.0
• Build.exe to MSBuild Handoff
• Automatically included in all project makefiles via \build\makefile.inc
• \build\makefiles.msbuild creates a response file (msbuild.rsp) in the
project directory and invokes msbuild.exe directly.
• MSBUILD_PROJECT in sources
NMake
Build.exe
Makefile.def
Makefile.inc
Makefile.msbuild
MSBuild.exe
My_Project.csproj
92. Lab Footprint (In-Market)
CoreXT - Build<Root>
Build.exe
• Windows build tool
• Orchestrator for building/traversing
the build graph
• Our version – 4.22.4093
• ‘dirs’ and ‘sources’
• i386 is default build target
Dirs
Dirs
Depth-first Traversal of Dirs
• SYNCHRONIZE_BLOCK/DRAIN
• Filter on platform tokens {i386},
{amd64}
Sources
Sources
Makefile.def
Makefile.def
MSBuild
WiX
Dirs
Sources
Sources
Makefile.def
Makefile.def
CMD
VC++
93. In-Market Branches
CoreXT - Build Contd.PASS[0|1|2]
• PASS0 – Generation of files, copying of static content
• PASS1 – C++ compilation
• PASS2 – C++ linking, managed code builds, setup/packaging, binplacing
• PASS2_BINPLACE – binplacing of built bits (mostly from PASS2)
Project Inclusion
• Integrating your project into CoreXT’s build
• Dirs, sources, makefile, makefile.inc, placefile
94. Servicing Concepts
CoreXT - mydirs• If a “mydirs” file is present in a
directory, build.exe will use it
instead of any “dirs” file to
determine what to traverse in
that directory.
• CoreXT build logic to
copy/rename “mydirs.<build
type>” to “mydirs” when <build
type> matches the current build
command.
Example: build layer
DIRS= \
Source \
<Root>
dirs
DIRS= \
Application \
Source
mydirs.layer
DIRS= \
Foo \
Bar \
Application
dirs
Foo
DIRS= \
Zoo \
Kernel
dirs
Bar
Zoo
95. CoreXT - Our Environment
CoreXT - Build typesBuild
type
Command
Pre-reqs
Description
Retail
Kernel
build retail-kernel
N/A
i386 build of kernel, frameworks and setup
Retail
AMD64
build retail-amd64 –amd64
Retail
kernel
AMD64 build of AOS, client, select other kernel
components, and setup
Layer
build layer
.product:<layer product>
Retail
kernel,
AMD64
Build of the specified layer product on top of the
kernel
Retail
test
build retail-test [*]
Retail
kernel
Build of the test frameworks and projects. With ‘*’
builds extended set of test projects (BATs, TAEs)
Debug
build debug-buildtracker
N/A
Build of the kernel and frameworks in debug
96. CoreXT - MSBuild
CoreXT - Layer Build• Builds the application component for a specific workload on a specific
layer
• Near 1:1 mapping of Product workload : Layer
• Exceptions – industry solutions in DAX6HF and DAX5SP1HF
• Settings for layer product: \build\productConfig.cmd
• Example: call %inetroot%\build\productConfig.cmd -Product
• Build command: build layer .product:<product>
• Example: build layer .product:FoundationSE
RUPayrollSE
97. CoreXT - Build
CoreXT - Layer Build Contd.Prepare base environment
Import/Compile Product
Extract Result
Static*
CombineXPOs
UpgradePreProcessingAX*
SetupDatabase
GUIImportXPOs
SetupApplication
SetupKernel
GUICompileAll
PostBuildValidation
CreateBaseEnv(SE)
GUICompileIL
DTDrill
ApplySEUpdates
DeployLocalizedLabels
DeployApplication
98. CoreXT - Build Contd.
CoreXT - Layer Build Contd.Step
Description
Static*
Unzips static content from RTM, CUs in checked-in zip files
SetupDatabase
Initialize the empty model and application database
instances
SetupKernel
Install the CD version of AOS and client
CreateBaseEnv(SE) Import any precompiled models that the layer product
we’re building depends on
ApplySEUpdates
Install any kernel hotfixes present on the CD
99. CoreXT - mydirs
CoreXT - Layer Build Contd.Step
Description
CombineXPOs
Combine all XPO files for the layer product we're building into a single XPO
to import
GUIImportXPOs
Import the combined XPO file into the AOT
GUIImportUpgradeXPOs Same as GUIImportXPOs but for the layer product upgrade model if
defined
DeployLabels
Copy ALD (label) files to the AOS install directory, restart AOS, run
labelsync
Import VS/SSAS projects Imports and compiles VS and SSAS projects to the AOT
GUIComileAll
Compile the full AOT
GUICompileIL
Generate the IL assembly representing the AOT
DeployLocalizedLabels
Import the localized ALD files into the localized label models
100. CoreXT - Build types
CoreXT - Layer Build Contd.Step
DeployApplication
UpgradePreProcessingAX*
SetupApplication
PostBuildValidation
DTDrill
Description
Take backup of DBs for bootstrap, export compiled
models for the layer product we're building
Copy the major version upgrade scripts to the CD
Copy application files to final setup location on the
drop (CD or SE Installer path)
Runs XPOVerifier against AOT to assert validity of
overlayering done
Export granular hotfix model(s) from the fully
compiled AOT using dependency data
101. CoreXT - Layer Build
CoreXT - VCSDef• MorphX version control settings defined in dynamically-generated file
%inetroot%\Definition\VCSDef.xml
• Defines source/label file locations for the models in the current layer
• Layer template location:
%inetroot%\build\ax_public\deployment\tools\vcsdef_<layer>.xml
• Template values expand from branch and product configs (e.g. label
ID ranges)
• Changing layers: SetStartupLayer <new layer>
102. CoreXT - Layer Build Contd.
CoreXT - Label Files• AX Label (.ald) files edited through the
label editor in MorphX.
• 1..* label files per layer product
• Located under
MorphX
User finishes editing labels in editor
and initiates submit
Label file locked
IDs allocated for new
labels
CoreXT
Sync/resolve label file
%inetroot%\source\application\labels
• LabelUtil command – validates
correctness of label files prior to
submitting
Run label file validator
Is label file
valid?
Yes
Direct submit label file
updates
No
Unlock label
file
103. CoreXT - Layer Build Contd.
CoreXT – Deploy.cmd• Installs the AX components onto a
developer’s machine and initializes the
developer environment
• Discovery of latest official build
• Initialize test environment (import of
framework and tests)
• Used for local developer and gated
check-in deployments
• Source:
\build\ax_public\deployment\...
Discover/copy build
Execute AXSetup.exe
Import test framework/
code
Checklist items / init
environment
104. CoreXT - Layer Build Contd.
Gates105. CoreXT - Layer Build Contd.
Gates - ExecutionMachine allocation/
Preparation
(Sync, nuke, uninstall old deployments,
apply changelist)
Changelist and
checkin
validations
Build
Deploy
Servicing Tasks
Test
Post Steps
(kernel, app, test, debug flavors)
(enlistment deploy for
UTs and BP checks, WTT
setup for BVT)
(Collect dependency data
on change)
(BVTs, UTs, BP checks)
(Determine Results, Submit CL, Send report email, copy logs,
update bugs, teardown and release machine)
(DynCop, SDDump macro
validators, code coverage
pairing , etc.)
Test Leg Machine
Build Leg Machine
Return results
Retail-test
(core)
Primary Gate Machine
Schedule
Run
Preparation
Retail-test *
(optional)
Deploy
BP
Checks
Public
API
Checks
Copy core test back to primary
Pre Checkin
Retail-kernel
Retail-amd64
Layer
Deploy
Pre-Submit
SDDump
Changelist
Tests
Unit Tests
(C#, X++)
Determine
Results/Submit
Schedule parallel leg builds
Build Leg Machine
Debug-buildtracker
Return results
Post-Submit
SDDump
End
106. CoreXT - VCSDef
Gates - FastKernel• Uses a pre-build kernel build to seed the gate process and skip the
retail-kernel and retail-amd64 builds
• Seed build share: \\dcsrdgtdfs01\KernelSeedDropShare
• Impact – can save over 60 minutes per checkin
• Invalidated by:
• One or more files not in whitelist:
build\wtt_checkin\FastBuild\FastKernelBuildWhitelist.txt
• One or more files in blacklist:
build\wtt_checkin\FastBuild\FastKernelBuildBlacklist.txt
107. CoreXT - Label Files
Gates - FastApp• Uses a pre-build application drop to replace the layer build done prior
to PreSubmitSDDump
• Seed builds come from the Rolling Hotfix BuildTracker jobs
• Impact – can save over 3 hours per checkin
• Invalidated by:
• One or more files not in whitelist:
build\wtt_checkin\FastBuild\FastAppBuildWhitelist.txt
• One or more files in blacklist:
build\wtt_checkin\FastBuild\FastAppBuildBlacklist.txt
108. CoreXT – Deploy.cmd
Gates - SDDump• Process used to capture the affected elements of a change
• Dependency data submitted to a central DB on DCSRDRPTSQLA
• Dependency Database per servicing branch: DependencyDB_<Branch>
• Data later used to generate a dependency bucket for hotfix packages
• SDDump Validation
• Macro (Pre-Checkin & Pre-Submit SDDump)
Changelist is unique
Changelist description contains bug ID
Bug is in the expected state and contains a valid KB number
Checkin layer is enabled for servicing
• Micro (Pre-Submit SDDump)
• Changes don’t result in deleted elements
• Changes don’t result in elements reverting to base layer logic
109. Gates
- SDDump Contd.Prepare base environment
Prepare Dependency Tools
Apply Changes
Extract Result
Apply Changelist
Run SERestore.cmd
Import XPOs / VS Projects
Remove SDDump Triggers
Run SEReplace.cmd
Import SDDump Validators
CompileAll
Extract SE model
Create SE model
CompileIL
Run pre-extract
(validators)
Import SDDump triggers
DBSync
Save dependency data to
file
Baseline build/deploy
• Create scripts to
revert/restore nonX++ changes
• Prepare environment
without changes
applied
• Restore X++ changes
• Import dependency
toolset
• Prepare models
where changed
elements will go
• Apply changes to AX
environment
• Execute checklist
items for fully
built/synced AOT
• Disable dependency
collection
• Validate changed
elements follow
servicing rules
• Extract model and
dependency data
110. Gates - Execution
Gates - Multi-product Check-in• Only in DAX6HF
• Checkin workflow to validate
changes to multiple industry
solutions in a single checkin
• Enables SYP and dependenent
overlayered chanes to IS
solutions to be submitted in
one change
• Primary machine does submit
and post-submit SDDump
111. Gates - FastKernel
BuildTracker Managed Builds112. Gates - FastApp
BuildTracker Builds - RollupPurpose
• Produce a full kernel + application build to the tip of the current depot state
• Checkpoint build to execute automations against
• BVT, BAT, Full X++, Full Best Practice, LMC
• Refresh localization binaries that subsequent hotfix builds will use
• Build type for Cumulative Update releases
• Produces update package for all changes since RTM
Cadence
• Full Rollup master with validation occurs ~3 times per week
• Runs daily when stabilizing a CU release
113. Gates - SDDump
BuildTracker Builds - Application HotfixPurpose
• Produce a granular hotfix containing the X++ change fixing a specific product
issue + any dependent change since RTM
• Builds Application against RTM kernel to assert no dependency taken on
binary hotfixes
Cadence
• On-demand.
• Auto-initiated in R3 branches on successful check-in. Otherwise manually
initiated by issue owner.
• Also produced as part of Rolling Hotfix schedule
114. Gates - SDDump Contd.
BuildTracker Builds - Binary HotfixPurpose
• Produce a cumulative build of the kernel and frameworks components
• Create patch installers (.msp) to apply fix to existing deployments
Cadence
• On-demand.
• Also produced as part of Rolling Hotfix schedule
115. Gates - Multi-product Check-in
BuildTracker Builds - JobsRollup Build
Binary Hotfix Build
Application Hotfix Build
Localization legs
Validate Parameters
Validate Parameters
Retail-kernel
Layer build
Retail-amd64
DTDrill
Code Sign CD1
Create Package
Patch
Validate Hotfix
Retail-kernel
Retail-amd64
Code Sign CD1
Translation-submit
Patch
Layer build
Retail-test-core
Retail-test-optional
Code Sign CD2
Create Package
Create Package
116. BuildTracker Managed Builds
UA Build117. BuildTracker Builds - Rollup
UABuild – Overview (Core Concepts)• UABuild – UABuild is our documentation build process
• Content is generated for word doc format, AX Help Server format, MSDN, and
Technet format.
• DXStudio – Content editor used by writers
• Content is store in DB: Axapta01 on DBServer: DCSRDUASQL01
• CoreXT/SD – Used for source control and to build content branch:
MainUA
• Get For Build – Process to pull content from DXStudio and check it
into Source Depot.
118. BuildTracker Builds - Application Hotfix
UABuild – Core Build Types• A GFB (Get For Build) pulls content from a DxStudio project into
Source Depot
• HXS Build – Builds content into HxS format, and optionally publishes
the compiled content to MTPS (MSDN or TechNet) using DxPublisher
• HelpServer -- Builds content into a format suitable for the Dynamics
AX Help Server
• DocBuild – Builds content into Doc, Chm, and/or PDF formats.
• ManagedReference -- Builds MRef content into HxS format.
119. BuildTracker Builds - Binary Hotfix
UABuild - Core Build FlowDXStudio
GFB – Based on
Project Name
Build is driven off
Environment Variables
for ProjectName,
Language, Forkname,
Target (Staging, Live).
Update Source
Depot Content
Folders For Project
UABUILD1-15
Build output to \\
dcsfrblddrop01\
MainUA\Build
\\dcsfrblddrop01\
MainUA\Build\
BaseDrop\
NoCheckin\EN-US\
Latest Content Pushed Here
120. BuildTracker Builds - Jobs
Daily Activities121. UA Build
Assessing Official Build State• Nightly builds scheduled, ran as expected, kicked off test runs
• Expected state - everything runs as scheduled and finishes in a timely manner
• Troubleshooting - For scheduling problems, check BuildTracker. For failures
and extended execution times, analyze the build logs.
• Evaluate build drop share resource check report
• Expected state - all drop shares report a healthy amount of free space
• Troubleshooting - if any drop shares show errors about low space (like the
one linked above)
• Run the bt.exe tool to get failed retention jobs and retry them. If they continue failing,
manually delete the drops
• If all retention jobs are passing, investigate if per-build size has increased or retention
policies have changed to result in more or larger builds on the drop. Log bugs accordingly
122. UABuild – Overview (Core Concepts)
Assessing BuddyBuild State• Check on buddy build queue
• Expected state - majority of the time the queue is shrinking or 0 and jobs are
completing (queue can be growing during peak times)
• Troubleshooting - Check on machine availability (e.g. are any offline). Check if
running jobs are experiencing hangs or other systemic issues causing them to
take longer
123. UABuild – Core Build Types
Assessing Gated Check-in State• Check queue length on MBSRPT
• Expected state - Queue as small as possible
• Troubleshooting
• Queue processor not running
• If queue is closed due to a break, work with owner of the issue to drive to resolution then
reopen the queue.
• Check if checkins are taking longer to complete or if there are fewer resources available to
schedule against
• Check running checkins
• Expected state - no checkins experiencing systemic issues resulting affecting the
execution of the gate workflow
• Troubleshooting – check WTT for machine states, check recent changes to
build\wtt_checkin
124. UABuild - Core Build Flow
Assessing Gated Check-in State contd.• Look through latest failures
• Expected state - All failures have been classified with incidents, all non-dev induced
incidents have bugs
• Check machine pools
• Expected state - All machines in the gate WTT pools are heartbeating and are in a
ready or running state
• Troubleshooting
• If the machine is ping-able and you can connect, log on and try cycling the WTTSvc windows
service. You may need to change the machine status in WTT studio to Unsafe if it has been
without a heartbeat for an extended time
• If you can't connect, log an issue through http://dcsrequest for someone in the lab to take a
look
• In either case, if a checkin job was interrupted when the machine lost heartbeat, be sure to
inform the owner and resubmit their checkin/abort the old one.