(AKA: Keys to Building Amazing Software)

Building amazing software doesn’t happen by accident. It’s the result of a disciplined team who understands the importance of key principals, or cornerstones of software. These key principles set the foundation for a successful piece of software. Software that’s both rewarding for the end users as well as the team responsible for building it.

In part one, we talked about the first two cornerstones of software that set the foundation needed for a project to succeed. We reiterated the importance of understanding the actual problem at hand. Additionally, we expressed how to intentionally focus resources on discovering and solving the problem.

Now, we need to talk about how those concepts meet the real world. How can you create a space with minimal distractions where the team can consistently produce valuable work? Additionally, how can you manage that backlog of work ensuring management of stakeholders’ expectations and delivering a complete solution at the end? Let’s start by creating an environment where features are built, tested, and delivered with minimal friction.

Cornerstone #3: Automate, Automate, Automate

In the initial ramp-up of a new project, we spend time on typical tasks like architecture and project set up. However, we also invest many resources up front to lay the foundation for what will eventually become a complex automated workflow. This allows us to more efficiently build, test, and deploy software. 

Within the entire lifecycle of a feature, writing code is a small piece of the puzzle. Testing the feature and how it interacts with the software ecosystem often consumes just as many resources than the actual technical implementation. This can also happen in the process of promoting a feature from a local environment to the eventual production environment. These are the most overlooked parts of the process, thus most inefficient.

As Phase 2, we often take the hardest parts and do them over and over until they become easy. It’s like exercising a muscle. The more you work it, the stronger it becomes. Regression testing is hard. Maintaining environments is hard. Releasing without downtime is hard. However, the more frequently you do those things, the easier they become.

That may sound good in theory. But, how can you hope to meet timeline and budget constraints while spending so much time inside the testing and release cycle? The answer for us is automation. 

It is often surprising for colleagues to learn Phase 2 doesn’t have a QA department. Nor do we have a DevOps team or layers of project managers to make sure the project stays on track. What we do have is experience with automating as much of the process as possible. We can focus on the implementation while still having confidence in all of the work that follows to bring a feature into production.

Automated Testing

Testing is one of the most important tenets of quality software. Yet, it’s often the first to go when timelines get tight. As soon as you open the door where automated testing loses priority, further delays and regression issues find their way in. That’s not to say that automated testing creates bug-free software. It does, however, prevent the same bugs from happening over and over.

We often follow the practice of both writing tests for new features and also for any bugs that may arise. If the bug makes it past our test suite the first time around, we know there’s a gap to fill in our testing. As teams prioritize this practice, the test suite grows over time to cover thousands of test cases. This leads to more confidence and better stability in our releases.

Infrastructure as Code

Most organizations have a process in place for source control. Source control ensures we have confidence in the current state of the code base as well as a way to revert back to certain points in history if the need arises. This level of control often ends at the code base. The rest of the infrastructure required to actually run this code is handled by less reliable processes. While automated testing allows us confidence in our produced code, that confidence is at risk if the environment that code runs on is out of our control. 

Machines maintain state. Sometimes that state is expected and other times it’s not. Software often expects the machine to be in a certain state and when it’s not, hard to diagnose bugs can appear. Having consistent, easily reproducible environments is critical.

A number of tools exist today for both cloud-based and on-prem infrastructure. These tools allow you to define everything needed for an environment from networking, storage, and operating systems to security, software, and any other necessary state. These environment definitions can be easily included in source control to ensure the same confidence and consistency as the rest of the code base. 

With infrastructure as code, environments can be treated like cattle, not pets. When an unexpected state arises in an environment, it becomes trivial to create a replacement. This eliminates the need to spend hours chasing down environment-related issues. It also makes the process of having production-like staging, test and development environments that much easier.

Continuous Delivery

Even with all of the above tools in place, the process of delivering a release to production can be error-prone and labor intensive. It’s not unusual for a release to involve running scripts on database servers, logging into application servers to copy code, and manually updating configuration files. Eliminate this manual process to every extent possible. Anything performed manually is an opportunity for error.

Continuous delivery is the practice of being able to release software to the production environment with the click of a button. You need no manual intervention other than the decision to release and the pressing of the button. All steps required to create and deploy the build artifacts happen automatically, including the process of taking backups, configuring the environment, and deploying the build and any dependencies it may have. We strongly believe the process for delivering software should be an enabler, not a blocker.

All of these automated processes work together to allow us to focus on solving the problem at hand while spending fewer cycles testing and delivering the solutions to the end users.

Cornerstone #4: The Iron Triangle

In every project there’s budget, timeline and scope. These are the three corners of the iron triangle (or a venn diagram as shown below) – our final cornerstone of software. In a perfect world, we’d know exactly what we will build, how long it will take, and how much it will cost. Reality doesn’t work that way, especially when building custom software. 

If you remember the previous post, I raised the idea of software being closer to the construction of a bridge where its dimensions and the laws of physics could change halfway through the project. 

Often, there’s a large set of unknowns up front. It’s like a hike through the mountains in the dark and your flashlight can only shine so far. We know approximately how high and how wide the mountain is, but the forest may be denser than we anticipated. We may discover a new path that takes us a different direction than originally planned. In software projects, priorities often shift.

While product owners use their best efforts to set targets for budget, timeline, and scope – as teams get into the meat of the project, they will adjust plans. When any one of these three points change, the other two will also change.

You Can Fight the Triangle, but the Triangle Always Wins

The Iron Triangle often isn’t fully understood until you earn a few battle scars. It’s a concept that comes with experience – the kind of experience that’s painful to get. You fight and even if it feels like you won, you later find yourself on the losing end of the war.

We’ve all seen more than one feature rushed through. There’s no time to extend the deadline, no other features cut from scope and no additional resources to account for the extra work. The feature is released – it may even be done on time – but it is half baked. Inevitably, related issues will continue to pop up months and years down the line in the form of bugs or workarounds. This consumes more resources than if it had been released right the first time.

They tried to fight the triangle. They added scope without adjusting the budget or timeline, but the triangle won. The feature was built, but it was more expensive and took longer than planned.

It’s critical all stakeholders understand the iron triangle. Many software projects fail before they begin due to this misunderstanding. Differing expectations are where software projects go to die. 

Those involved with building the product, providing resources for the product, or are expecting to be end-users of the product are all concerned with budget, timeline, and scope. However, each will likely prioritize different areas. Setting realistic expectations for all stakeholders, and communicating constantly about the balance of these three competing factors throughout the project, is the only way to ensure all parties are on the same page. This goes a long way towards the overall success of everyone involved.

The Iron Triangle isn’t just a vague project management term from the past. It’s a critical model of how the real world builds (or tries) things. It’s a tenant that everyone involved in the project must understand or all of the work we’ve put into the other three cornerstones goes to waste. We need all four corners in order for our foundation to stand firm.

Building Amazing Software

Like any large engineering project, success doesn’t come because you got lucky or happened to make a few good decisions or you made some bad decisions and then scrambled and pulled it all together in the end. It’s not something that happens by accident. Success comes from having a well-thought out plan based on a solid foundation. 

In software, that foundation is constructed from these four cornerstones. It’s these non-negotiables that form the base upon which a software project can succeed. These cornerstones are what allow the project to withstand the obstacles that always come in a large engineering effort. Understanding your core problem, building with an agile mindset, prioritizing automation, and managing the expectations of the stakeholders throughout the project are what leads to success.

If there is a secret sauce to building great software, these would be at the top of the ingredient list.