The evolution of DevOps principals is the next step in the story, but before we get there, we need to understand what changed to make some enterprises make the leap to agile.

One of the biggest complaints I have heard has been around application performance, stability and the number of bugs end users find in production with B2B customers questioning why they are paying vast sums of money, to ‘test’ the software while it’s in production. This factor led one customer to explore Agile development in order to get a handle on improving application quality.

At around the same time, as applications became more sophisticated, the consistency of the underlying infrastructure was becoming an issue. In order to scale production or to quickly test new features in isolation, creating new instances of the server environments in a timely manner was becoming necessary. Enterprise customers consistently had issues with both provisioning new environments and maintaining the correct configuration settings and app dependencies. Luckily, this was also at the time when the concept of infrastructure as code and automated provisioning was becoming popular, so customers were making some inroads into agile concepts and automation for reasons other than adopting Agile practices.

In order to be successful with agile practices, teams realized that they should adopt a truly Agile approach. A decade ago, one of the services that we offered was a lightweight development practice analysis which measured the maturity of a developer organization at a very high approximation. If necessary, this was followed by an in-depth Agile assessment. Having delivered the results of the in-depth Agile assessment, customers would realize the gaps and choose a model to work with. Scaled Scrum was a popular choice of Agile discipline adopted by many enterprise customers and their adoption of more “Agile” approaches spurred on the development higher quality code and more stable applications. Realization that certain aspects of the application lifecycle should be considered earlier in the lifecycle forced customers to include security best practice checks, unit testing and operational provisioning in order to make them more successful. So, why wasn’t everyone embracing these new practices?

Blockers

Convincing the business that this is the way forward was the key to success. Adopting a DevOps culture is a huge shift in the way applications are developed, not just for the development teams, but the layers around and above that support them. Test, QA, Operations, Security, image provisioning, etc., all need to be uprooted and re-included. Not only did the business leaders needed convincing, but often their B2B partners/customers needed convincing too because the possibility of features not making a given release in favor of higher quality, became a reality. Promises of fewer bugs, better change control more opportunity to provide feedback and a steady flow of new features where often enough to convince the business this was the way ahead. The possibility of new features on a regular cadence often scared not only the business, but the dev and IT teams too. The core tenant of a successful Agile lifecycle is the delivery of value to the user (in the form of stable high-quality working code) on a continual basis. The delivery of value to the end user is adopted by many teams as a key part of the Definition of Done, blowing the minds of business owners and some IT leads since many organizations tend to wrap a mini waterfall process with Agile ceremonies delivering working code at pre-determined milestones (quarterly or semi-annually.) This is often referred to as “Scrum-fall” and besides having a piece of software that can be demoed at the end of each sprint, the real hardening takes place much later in the lifecycle as the code still has to be “thrown over the fence” for security and compliance checks, testing and QA etc.

Another blocker is the term CD as in “Continuous Delivery” which is often misunderstood, especially when we talk about CI/CD pipelines. CI/CD pipelines are the basis of any modern Agile and DevOps development approach. To many, CD is interpreted as constant deployment (to production,) whereas, we are really talking about delivery of tested software with minimal bugs, that can be integrated into the main working branch and delivered to a repository. Feature flags or a similar mechanism in place allow this to be extended to deployment to production. Where the concept causes trepidation is when the customer is used to deploying once a quarter to production to meet a milestone.

As implied by the last paragraph, the next step for a typical enterprise is to implement CI/CD and this is often rejoiced and celebrated by the deployment of bits into various new environments from Dev to Test to QA and sometimes pre-production if the operations team will allow it, but hardly ever into production. The customers I have worked with have eventually embraced their Operations teams with the idea that automation from the same CI/CD pipelines can also safely deploy into production. Shifting a mindset (whether it’s Ops, Sec, or others who are fighting and screaming) into a DevOps culture is the hardest aspect. While hands on UI testing has a place, whether as part of a ring-based deployment model (which is way beyond were we are in this discussion) or through failing fast, automated testing has an extremely important place in this discussion. The realization that automated testing is difficult often comes late into the realization that getting to an advanced Agile or DevOps state is rather difficult. Shifting left, not only means writing as many automated tests as possible but writing tests that have very few dependencies. At Microsoft we label these dependencies (starting on the far left) as L0, with no dependencies, L1 with one dependency etc. and increasing as we move back to the right. In order to do this successfully, it is often necessary to rewrite code in order to make it testable. This has a high cost and really stretches organizational sensibilities when it comes to being Agile – as discussed above, the business and PMO don’t often consider the cost of technical debt and NFRs when they explore the cost of developing an application. As the backlog of new features continues to grow, it’s hard to rationalize such costs. On the other hand, without automation and the appropriate pipelines in place, it’s hard to envision rapid deployment of continuous value to the end users.

How a DevOps Culture can help

Having touched on many of the blockers and steps that organizations often take to move forward with an Agile culture adoption and then on to a DevOps culture, it might be useful to discover what steps we, as Premier Developers, go through to help our customers move forward. First, we strive to understand the pain points that the customer is suffering. For example, if waterfall or CMMI is working just fine and they are successfully deploying high quality applications and the end user sees value, then perhaps there is nothing to do. However, if we hear that the end users are opening tickets and the developers are unable to deploy fixes or new features without significant delay or downtime, we move forward. For a while we would conduct workshops on “The Day in the Life of a Development Team,” or “A Release in the life of an Agile Team”, and while these were often eye opening for the customer, they often where too broad in their scope and coverage so they didn’t meet the needs of everyone in the audience. Now, we usually begin with understanding the current state of application development at a customer with a thorough DevOps assessment. Agile and DevOps assessments consist of an engagement that covers about 2 weeks; during the first week we introduce what the purpose of the engagement is, conduct multiple interviews (usually face-to-face) with individual representatives from each role that encompass the complete lifecycle of the application. This hopefully crosses all the disciplines including dev, sec, ops, business ownership, etc. Finally, we deliver a quick high-level review of the preliminary results before the final analysis and delivery of the results.

The outcomes of the assessment have changed little over the past decade. Ranking the maturity across many different areas across people, process and technology so we can understand the overall environment and then stepping back and making a set of recommendations of next steps (usually identifying low hanging fruit) on how to move forward with a plan of prioritized actions. How we approach helping a customer move forward has matured over the years. Possible actions range from helping form teams and giving advice to on how to operate those teams, to helping set up and manage a backlog etc. However, based on the outcome of an assessment, what we tend towards today, as a viable next step is to explain how product groups at Microsoft have transformed from non-Agile through the Agile adoption process and onto a DevOps culture. These discussions and education sessions are conducted at various levels and with different groups, from the CXO stake holders, down the chain to the development teams. At one point we would sometimes have customers visit the PG for discussions on how a particular team worked, but this sometimes lead to the adoption of the wrong behaviors, so we tend toward a customized (depending upon the attendees) delivery stressing that they should learn from what we share rather than copy what we share.

Moving beyond this step really depends on the size of the organization and how we feel we can best help them. We often include Microsoft Consulting Services to do custom engagements where they are embedded in a team and help with all aspects of DevOps, including designing pipelines, writing test code, reinforcing the culture etc. A relatively new and recent MCS offering is to invite the team to take part in a DevOps DoJo. Here the team is immersed in a sprint long engagement with DevOps experts overseeing every step of the way.

Conclusions

To wrap up. I’ve not covered some of the key elements of DevOps culture, like the concepts of sharing code and inner sourcing etc. which comes as we move DevOps practices across an organization, but generally, most applications are built in isolation from one another; code and proven practices are developed multiple times so that common code and solution architectures have been repeatedly reinvented. As customers automate their processes, they generally need to shift their culture to accommodate the changes throughout and to minimize bottlenecks. Adoption of Agile processes, the introduction of pipelines, shifting left (and right) and failing early are all prerequisites that lead to an easier adoption of a DevOps culture.

As customers move to Azure, sharing best practices across the enterprise becomes more important and reshaping their organizational structure and transforming them into modern Agile practices with a DevOps culture, while quite difficult, will lead to the faster delivery of stable high-quality applications with fewer bugs, less technical debt the ability to more rapidly deliver value to the end users.