Problem
Suppose, Ahmed is a developer who writes code for new products, new features, bug fixes, and security updates in the products. Ahmed is working for a company whose success is dependent upon offering new and exciting products to its online customers faster than its competitor. Unfortunately, Ahmed needs to wait weeks for his work to be placed in the production. And, this delay increases the pressure of staying competitive because some competitors are able to deploy new products and new features much faster. In addition, this delay makes it very challenging for the developers to manage both, the code that is pending to be pushed for production and the development for the next feature they are responsible for producing.
And when Ahmed's code is deployed in the production environment, occasionally unseen errors and problems occur. It mostly happens because Ahmed was just focusing on writing the code based on the development environment and it is not identical to the production environment.
Now, the Operations team comes into the picture. Zunair joined the operations team and he is responsible for the 99.9% uptime of the production servers. And as the number of products developed by the organization is continuously increasing, it is very difficult for Zunair to maintain all the servers. This is because the tools which he is using to administer all these servers are not so much effective. This creates the challenge for Zunair to migrate the new code into the production environment. And once the new code is deployed, it is his responsibility to monitor all the problems and errors occurred by this new change into the production. Zunair has to do all the SMOKE tests on each build into the production.
Sometimes, it also happens in the scenarios, when the teams have to manage 4 different environments.
The Quality Assurance Engineer has to do all kinds of Smoke tests in the application in different environments, which takes a lot of time and efforts of the complete team. Sometimes, there are a lot of problems in the shipments and QAs have to identify all kind of problems by running those shipments in environments. Suppose, if there is something wrong in QA environment by running that shipment, then QA reports the issue to the development team and then, developers have to solve that issue with new efforts. When the developers are sure that now things are working fine, then it will be run on the QA environment. After passing the Smoke test on the QA environment, the quality engineer runs the shipment on UAT (User Acceptance Testing) and if the things are working fine then it goes to production otherwise the issues are reported back to the development team.
This is how it takes a lot of time and efforts on the part of the team.
Time Consumption
We can conclude from the above example how much time will be consumed to move the things into production. Tracking bugs is always very important in software development. Earlier, we tracked bugs in the product which consumes less time. The bugs which we track in the code during development is the best condition for saving time.
Now suppose, this buggy code goes to the QA environment. The QA or Test engineers find out bugs in the code and report back to the developers. In this scenario, it consumes time more than the first scenario.
Now suppose, if the bug in the code is not found out by the Test Engineers and the bug goes into production. The customers report the error and strange behavior of the application. And then, the client escalates back to the product team. In this scenario, it will consume a lot of time and efforts of the team.
What is DevOps?
Here, DevOps comes into the picture to resolve all of the above problems. DevOps itself is not the name of any software tools. It is basically the name of the culture. It is the name of a mindset to do the work. There are many definitions of DevOps on the internet but I would like to quote only 2 clear definitions.
DevOps is the union of people, process, and product to enable continuous delivery of values to our end user. ~ Donavan Brown
DevOps integrates developers and operation teams in order to improve collaboration and productivity by automating infrastructure, automating workflows, and continuous measuring application performance.
If you read these 2 definitions one after the other, then the things will automatically get clear to you. DevOps helps both, developers and operations team, to make the product mature. We always look for the solutions and strategies on how to deliver our product very quickly to the customers.
And to make things faster, engineers started looking into the automation because it is the only way which can make things smarter and save a lot of human efforts and time.
Traditional vs Modern Methodologies
You might have noticed that in our industry, many organizations are moving towards Agile way of working. This means that the waterfall approach we have embraced in the past decades is now fading. We are in the major paradigm shift; and, we are moving in an engineering way of thinking about software delivery towards a manufacturing approach.
Here are some of the points to consider while learning Traditional vs Modern Methodologies in Software Engineering
|
Traditional
|
Modern
|
|
Waterfall Approach
|
Agile Approach
|
|
Different teams or organizational units for requirements, development, test, and operations
|
Multidisciplinary teams where all disciplines work together on a small piece to deliver
|
|
Clear separation between Business and IT (demand/supply)
|
Business, Development, and Operations in 1 team
|
|
Release software 2 or 3 times a year
|
Release multiple times a day
|
|
Budget/cost driven
|
Value stream driven |
So, first of all, let’s discuss these differences before moving forward. In the Software Industry, the traditional way is that we collect all the requirements at a time and then, we follow the complete SDLC. Agile is the way in which iterations and incremental approaches come into play. That’s why, in Agile, the requirement changes frequently and changes are always welcome. In Agile, the highest priority is to satisfy the customer.
In the traditional approach, there are different teams for requirement engineers, development teams, testing teams. And in modern methodology of software development, the customer is directly linked with the developer to decrease the communication gap and to make the product according to the real and actual requirements of the customer.
In a traditional approach, there is a clear separation between businesses team who are gathering requirements and the IT team whose responsibility is to supply the product. While in the modern approach, all these things are linked together in one place.
In the classical and older traditional approach, only 2 or 3 software releases are successfully released but in a modern approach, we can release features multiple times a day.
So, it is important for all of us that we need to follow Agile methodologies and try to think out of the box to make things work faster.
Agile and DevOps
Most of the times, people are confused and they think that DevOps is a new thing while sometimes, they think Agile and DevOps the same. Actually, as I’ve already mentioned, DevOps is not a new thing. If we read the Agile Manifesto, we can get an idea behind Agile and DevOps.
“Our highest priority is to satisfy the customers through early and continuous delivery (CD) of valuable software.” (Agile Manifesto)
So, it is concluded from the manifesto that when the Agile movement is started, they already explicitly stated the continuous delivery of valuable software. And continuous delivery is all about creating repeatable, reliable processes to deliver valuable software to our customers very fastly. And we have to automate nearly everything in order to achieve Continuous Delivery.
But still, Agile and DevOps are two different things. Agile basically fills the gap between business/requirement engineers and developers.
DevOps fills the gap between Software Developer and IT Operations team.
DevOps PlayBook
In Agile, there are predefined methodologies like Scrum and XP which are used to implement the agile in the organization. Similarly, for DevOps, there are not very well established but most common methodologies that are mentioned below.
People/Process/Time
In this methodology, we identify the group of people who are responsible for the job functions, then we define the processes implementation by those people and then we specify the tools. It might seem to you very obvious but sometimes engineers and tech managers do the reverse steps and buy the tool first.
Continuous Deployment
It is just the practice of coding, testing, and releasing software frequently in the form of very small batches to make the quality much better.
Continuous Integration is the practice of automatic building, unit testing the entire application, and ideally, it happens on every source code check-in. Continuous Delivery is the practice of deploying every change into production and performing automatic and integration testing on the code. Continuous Deployment is the practice of automatic deployment into production.
Most of the times, all 3 above terms are used for the same concept. There is not so much a huge difference between them. And if we dive little bit deep into Continuous Delivery, you’ll see the bigger picture here.
With the help of source control, after we check-in the code, the code builds in the pipeline and all the associated unit tests run with that build. If all the tests are passes and code validation is OK, then it makes the package of the complete build and forms into artifacts. I’ll explain the artifact later.
Best Practices
When you’re making the build, there are some important points which we need to care for.
- Build should pass in less than 5 min
The longer build takes, the more people naturally wait until they have large batches of changes which increase your work in progress.
- Always commit small changes
- Don’t leave the build broken
If the test cases are failed or code is not compilable then the build will be broken and don’t leave the build broken. If the build is broken, you block delivery. Stop all the meetings and stop all the team tasks until the build is fixed.
- Don’t allow strange tests
You might have experience that someone writes the code which sometimes works and sometimes doesn’t work. So, there is a problem in the code. Don’t write such kind of tests.
Once you have planned out your CD pipeline, always track changes. And ask 2 questions to yourself.
Are you able to track changes in the whole system? (Cycle)
How fast you can move your changes from development to production? (Cycle Time)
So we should have the matrices, how long it takes the time to flow the code from dev to production. It is also important because if you read the definition of DevOps, it is also the duty of DevOps engineer to measure the matrices and performance of the application. And try to make it better.
Lean Management
Lean Manufacturing or Lean Production, often simply “lean” is a systematic method for waste minimization into the production system without sacrificing system productivity. Japanese are very progressive to make things productive. Lean technique basically introduced from the Toyota Production Company. The main purpose of the Lean is to make things more productive and in less time and reduce the waste from the product.
Let’s discuss some of the techniques followed by Toyota Productions to make things more productive.
Andon Card
It is a Lean manufacturing principal. In this technique what happens is, in Toyota productions if a single employee feels any kind of bug or waste in the production then he try to remove that bug and if there are chances that this waste can move into the product. He/She pulls the Andon card. After pulling Andon card, complete production stops and then all the factory tries to remove that severe issue which can be very harmful for the company reputation.
According to Toyota Finance, after pulling the Andon card a company loses $1 million. But if that waste or bug moves into production and into products, then the company may lose hundreds of million dollars. That’s why the Andon card technique is very important to make sure of the quality in the products.
Blameless Postmortem
System always matures after continuous failures. If you want to make things more productive and powerful, then it is necessary to face the failures. Because failures help us to learn more and to do more. Humans are not wrong but processes are wrong. In Lean management, one of the best practices is if there is any kind of failure occur in the system due to someone; instead of blaming him and demotivate him. The complete team does the blameless postmortem of the incident. They log everything what happens and how happens and then all of the team learn from this incident. This is how they respect each other and make the team environment much better.
Kaizen (Continuous Improvement)
In this technique, teams are mostly focusing on different kind of experimentation and learning. They share ideas and problems with each other. They are actively participating in the things which are working and which are not working for them in their scenarios. This is how they master the skills by picking up and trying new things.
Lean Management is strictly following in the Toyota Production System but now it is also implementing in the Software Industry. If you want to know how can we implement the Lean techniques; here is a detailed book for you (Lean Software Development (Agile Toolkit))
Blue / Green Deployment
If you read the different types of deployment models, one of the ways is the Blue / Green deployment model.
Actually, what happens in our real life, different teams are working on the same product. And this is the modern era in which different batches of commits or I would say different releases of software happens in a single day. Then it would be very difficult to track which changes affect our complete system. So there is a model of the Blue/Green Model.
In this model what happens is, we have 2 types of servers. 1 is a live server (blue) and the 2nd is staging server (green). Initially, traffic goes to the live servers and when we need to release a bunch of features then we deploy those new feature changes to our green servers.
And then we run the smoke tests to the green servers to check if everything is working fine. And when all the test passes we route our traffic to our green servers. And when there is any kind of error in the green servers, traffic will be routed back to the blue servers. This is how we update our changes in the application servers.
Underlying persistence layer (database) is the same behind both of the servers. This is how blue-green deployment works and help us to track if there is any kind of error in the production and we can easily revert our changes.
Chaos Monkey
As we already know that it is also the responsibility of DevOps engineer to keep track of the performance of the system and measure the matrices about uptime and downtime of the system.
Netflix is a media service provider whose complete system is based on modern technologies. It is using AWS (Amazon Web Services) cloud to provide the media services to our customers.
So you might be thinking what is Chaos Monkey?
Netflix engineers wrote the script which is used to affect the production servers. This code is open source and available on the GitHub. Now, when the chaos monkey is affecting the servers, Netflix Engineers fight against this monkey disasters in the production and keep the servers up and running. Chaos Monkey randomly shut down the servers. This is how Netflix Engineers test the system from unexpected outages and unexpected failure conditions.
As I have mentioned above in this article, if we want to make the system successful then we have to face failures and fight against those failures. And if we are not facing the problems then there are very few chances to be successful. Humans are not wrong but processes are wrong. So we have to make our processes strong.
Automation
As we have already discussed, DevOps is the name of the delivery of valuable products to the customers very quickly. And if we want to do things fast; then we have to automate our processes. And here the Automation comes into play.
With the help of automation, we can automate things. Here, we have some examples of automation -
- Infrastructure as a Code (IaaC)
- Automated Testing (Unit Testing)
Infrastructure as a Code
IaaC is a technique through which we can provision the new resources programmatically and not even we can provision the new resources but all also we can maintain and configure the new resources.
If you have some experience with any Cloud (Azure, AWS), when we are creating the resources then we can see the IaaC templates that are used to provide the resources. Like Azure has an ARM and AWS has its own. So these JSON templates are used to provision the new resources. This is how IaaC is used to help us in our real life.
When we are working with DevOps, we need to use the servers at different point of times and you might have experience when we are working with new servers and deploy our application. Most of the time our application is broken down because our new acquire servers have different configurations. If there are less number of servers like 2 or 3 then we can configure them manually but suppose you’re working enterprise application and we have to configure hundreds and thousands of servers at a time. So we have only option to use the IaaC approach to configure all the servers within a few minutes.
Cheff, Puppet, Ansible are examples of IaaC automation and they are mostly used for the configuration management. We’ll explore all these tools in our upcoming series.
Automated Testing
So automated testing is the second name of unit testing. When we are following the DevOps coding practice, we have to write the unit tests. We commit the code with its proper unit test cases. And then in DevOps pipelines, our tests run and make the build. This is how we automatically test our application with different kind of scenarios. We can write the Unit test and we can also write the behavior driven tests of our application.
This is how automation helps us in DevOps.
CI Toolchain
If we discuss the open source tools of DevOps, there are many tools in the market. And different organizations are using different tools and even some of the organizations wrote their own tools to implement a different kind of operations.
If we talk about the dashboards or boards where we manage the sprints and backlogs of our project, then we have the best option JIRA and Trello. And if we talk about to run the test and make the build of our software product then we will have the best option of Jenkins which is famous and popular in the market. If we talk about the version control systems then there is no competitor of Github which is used to maintain the public repositories and if want to maintain the on-premises source control then we have the options of GitLab and BitBucket. And if we want to deploy the featured product then we have the option of XebiaLabs Deploy and Octopus Deploy.
There are many other tools which I’ve not mentioned yet. And all these tools are amazing.
Azure DevOps
Azure DevOps is a SaaS (Software as a Service) kind of thing provided by Microsoft. All the above features like boards, testing, version control, deployment, and making artifacts of the software product. Everything we can do with Azure DevOps. Azure DevOps is the collection of all kinds of these features.
Azure DevOps Boards where you can collaborate with each other on the work that needs to be done to create the software product. Boards are focusing on providing a collaborative environment that fits the need of the team that is working according to an Agile and DevOps way of work.
Azure DevOps Repos are used for the purpose of source control. Here, we have 2 options to use the Git and TFVC. And we can commit and track all our changes to Repos.
Azure DevOps Pipeline provides us the way to do the CI builds for any platforms and low-level language. It provides a web interface where we can create jobs to create production artifacts for your software.
There are 2 kinds of pipelines.
- Build pipelines - which are used to build the software product.
- Deployment pipeline - which is used to deploy the software to any target platform (GCP, AWS, Azure)
Azure Devops Test Plans provides you the way to fully integrate your tests into your agile way. We can track requirements on the tests. We can run exploratory tests against the specific deployment that comes from the deployment pipelines.
Azure DevOps Artifacts are just like the version of the software. You can upgrade the artifact with its latest version which is built by the pipeline and if you find any kind of defect then you can easily revert it back to the previous artifact.
Another advantage of artifact is that if you’re using any node package in your software and this node package has some kind of bug inside which is affecting some of our product features and we delete the package from the artifact and next time when we’ll install it again, then artifact automatically take care of this and download and install the previous version of node package which we installed before. That’s the beauty of DevOps Artifacts.
Note
The most beautiful thing of Azure Devops is that we can use Azure Devops to maintain our product and we can even deploy our application on any platform like (Azure, AWS, GCP, IBM) etc. It is totally dependent upon the client choice. And if we want only few tools in this complete suit of Devops tools then you can use that specific tool only and only pay what you use.
Conclusion
DevOps team tries to automate everything from testing of new code to infrastructure in provision. In DevOps practice, we write the software in small chunks instead of going live in 1 go with all the features. In the traditional approach, we write the bulk of the code and consume months of testing on it. Writing small chunks of code allows us to increase the frequency of deployment and improve the time to deploy new code. It also enables us to monitor and measure the iterative process to improve the code and operations every day. Instead of configuring software and hardware resources manually, we can use the DevOps tools to configure our systems within a few minutes. This is how we can save huge time for the teams in the company. With the help of DevOps, we are mainly focusing on the business instead of doing all the things.