The Problem with
Traditional Software Development

A fundamental truth about modern business is that how we collect and utilize data has a profound impact to our success or failure. In turn, the code that we create and maintain to manage our data becomes a critical factor for our business success.

What is the problem with our traditional methods of building complex business systems? Many organisations are running on legacy systems that are barely maintainable. They spend large amounts of time and resources patching fragile integrations between obsolete and sometimes unsupported applications. Adding new functionality, even a few extra fields, can be a major project and present a significant cost and risk to the business.

We are constantly trying to address the software development problems with new technology, architectural patterns, and DevOps processes. The key question is: are we addressing the right problem? Going forward a strong guiding principle we should follow is “Functionality is an asset, code is a liability” a thought which Kirk Pepperdine tweeted to Kelvin Henney. In other words, writing code is a “necessary evil” to create the functionality that ultimately delivers business value. Having a lot of code requires a lot of effort to maintain and secure it.

Maybe the question we should be asking is whether we are writing too much code to get the required functionality? Maybe we can get the same or more functionality whilst writing less code? Maybe we could write simpler and less complex code? Or even better - can we do away with the code altogether?
Let’s start by thinking about what we mean by functionality and code.

Functionality is an Asset, Code is a Liability


Functionality is an Asset

In a perfect world Information Technology would be invisible. It would:

  • always be available,
  • present the same data on any channel,
  • do exactly what the business needs today,
  • change effortlessly as the needs of the business evolve,
  • drive business growth and generate new opportunities, and
  • be secure and provide privacy for our customers.

Unfortunately, functionality comes at a price – we need to create code to provide it.

Code is a liability

Let’s think about code in its broadest sense:

  • The documentation we use to specify, design, implement, and test a business solution
  • The source code we write to provide the functionality
  • The test cases we write to verify that our code works
  • The servers and services upon which we deploy the solution

They cost us when we create them initially, as they evolve over the course of the initial development project, and as they adapt to reflect the changes in the business over the solution’s life cycle.

What is the Traditional Solution Development approach?


At the start of a new software project we try to determine the scope of what will need to be developed by attempting to define the requirements in the following areas:

  • The data model we will need to support the functionality.
  • The user experience (UX) that will expose the functionality.
  • The actual functionality that will be automated within the solution.
  • How data is exchanged with other internal solutions and external third-party applications.

Now for most projects, our initial requirements will be high-level and somewhat ambiguous. As the project evolves the requirements will be refined and, in many cases, redefined. This leads to a sequence of change management activities and re-baselining the project plan.

To illustrate the typical solution development process, let’s consider a “simple” solution that will gather data for an insurance policy management system that will gather data for a quotation, rate, and issue an insurance policy. For clarity, we might decide to break this solution into two micro-services: Insurance Policy and Quotation.

Let us begin with Insurance Policy micro-service and the concept of the Policy itself – we will need a few attributes:

  • Policy Number
  • Effective Date Range
  • Legal Name of Policy Holder
  • Address of Policy Holder
  • Etc…

We can define the names of the fields we need, indicate their data types and some validation rules… At this point we have enough to begin our database design and we can create our first table – progress has been made!

So now we can add some more concepts such as Coverage, Exclusion, Wording, Beneficiary which will be children of the Policy and enclosed by the Insurance Policy micro-service. Our data model is now emerging, and we can start to think about the rating functionality and the data we are going to need to create the finished Insurance Policy.

We can now start to consider the user experience for our Insurance Policy micro-service. We could consider a back-office user role for the broker that would allow them to examine and modify the Policy data. We might want to build this user experience as a web application over a database – we break out a set of user stories and assign to our agile project team to build.

If we are working in a mature development environment we might already have a standard architectural framework and DevOps processes in place. If not, we are going to need to set some baseline standards and processes before our teams can get started.

Typically, we would now expect our DevOps team to begin creating a database with a table for each concept. Using a standard Create, Read, Update, and Delete (CRUD) model, our team would now begin to create a set of stored procedures to maintain the underlying tables. So, for each concept we now have 1 table and at least 4 stored procedures (we might end up creating extra stored procedures for specialized searches and partial record updates).

The team would then go on to create a set of business objects (one per concept) to call the stored procedures and provide an abstraction layer over the storage for the user experience.

At this point we may also start the creation of our automated unit tests at the database and business object level. The size and complexity of the unit test software we will need to create can easily be of the same order of magnitude -or larger- than the actual functional code we will be testing.

Once we have some business objects, either actual or test stubs we can begin to create the user experience. In this case we are going to have to create multiple forms for each concept, we need to be able to:

  • search, find and select instances of the concept
  • display and possibly edit the attributes of the instance
  • invoke automation on the instance (e.g. rate the policy) – that’s surely just a button on the details form that has some code behind it 😊 no big deal!

Now all that’s left is to add some automated test scripts for the user experience and we are done, right?

At this point we can easily be up to 20-30 tables with a couple of hundred attributes / columns, 100+ stored procedures, 20-30 business objects with 100+ functions, and 50-60 forms in our user experience. We could easily have about 100,000 lines of actual functional code and another 100,000 lines of test harnesses and scripts. Taking a conservative cost of $1/line our project is already up to $200K+ just to cover the basic data model and a simple user experience for our brokers.
Now the fun begins!

What about security?We need to ensure that not everybody has access to the Insurance Policy data. We are going to need some mechanism for securing access to our micro-service and restricting the policies visible to a user. This implies we need to add some extra tables to map users to policies and then we are going to need some more stored procedures, business objects, user experience forms and testing software. Hopefully, we find this out towards the beginning of the project – security is better if it is built into the design from the beginning and not added as an afterthought!


What about audit? Increasingly, we are seeing system requirements to track changes to records over time. This implies that if we have 20 tables of primary policy data we need to add another 20 tables to track changes to the data – one per primary table. We can add a version number to the primary data and a database trigger that copies the new row from the primary table into the associated version history table. We will also need some new read stored procedures to pull specified versions of the instance from the version history tables. We typically don’t need to create extra business objects or user experience to handle this data – we could just add a little extra complexity to request a previous version of the record and ensure that the user can display it but not edit it or perform actions against it.

We have now added an extra $50K-$100K of software to implement security and audit – we must be done now, right? Well this gets us a simple Insurance Policy micro-service into UAT and our business users are now going to get immersed into testing it and figuring out what they missed in the initial requirements. This is where we find out:

We missed an attribute on the Policy object – no big deal we just add a column to the Policy table, the Policy History table, the trigger that creates the Policy History, all of the CRUD stored procedures used to maintain Policy/Policy History data, the business object the user experience forms, and all of the test harnesses and scripts. Perhaps it will take a day to get it all done and everything back through the DevOps Continuous Integration / Continuous Deployment (CI/CD) pipeline? Unfortunately, it’s much more expensive and risky to add the field now that we have built the UX.

“We need to redesign this form – I don’t like where these fields are located.” At last an easy one; we just need to change the form layout, tab ordering, and update test scripts.

Wait – this only works for Home Insurance Policies! We just expanded our business into Vehicle Insurance Policies. Do we need a whole new micro-service from scratch, can we clone and change the existing micro-service, or can we expand the micro-service to deal with a new type of Insurance Policy?

Our costs are now really starting to mount up. Small changes to concepts are taking longer to implement and a big request like a new type of policy could trigger a whole new project. If we are lucky, we will trap these issues during User Acceptance Test (UAT). If we are unlucky, we will already be in production. When you consider that our micro-service may be in production for many years these enhancements become a near certainty and now we must deal with the impacts of data model changes on production data. Just adding a field becomes much more difficult, expensive, and fraught with real business risk. In many businesses it becomes increasingly difficult to get even simple changes though and the ability of IT to support the business diminishes over time until IT becomes an anchor around the neck of the business.

Why development projects fail?


Scope and Requirements

Getting perfect requirements at the start of the project rarely happens! Usually, our requirements evolve as we show the real business users the solution in User Acceptance Testing (UAT) or production. As an industry, we have begun to embrace agile project management methods to begin to address this risk and we have certainly had a lot of success compared to the older waterfall models. We are now failing faster but “fuzzy” requirements are still the biggest risk to ultimate project success.

Lack of Architectural Vision

Many organizations allow their solutions to grow “organically” without a pre-defined architectural framework in place. This “organic” architectural approach is not the same as “Evolutionary” architecture. In the organic approach new technology is added piecemeal with no ultimate end state in mind. Projects based on organic architectures tend to be focused on short term tactical solutions with the result being trying to build a jigsaw puzzle where all the pieces are blank, and you don’t have the picture! On the other hand, evolutionary architectures have a desired end state in mind and new technology is introduced to move the solution towards that end.

Proliferation of Technology

When we create a new micro-service we should use the “best” technology for the job, right? Usually this comes down to the personal preference of the development team tasked with building the service. Most services could be equally well implemented in C++, Java, PHP, C#, VB – just pick your preferred technology, language, and development model / library! This leads to the adoption of many different technologies in use within an IT organization and causes a lot of down stream problems with sustainability over the longer-term operations life-cycle.

Too much code

Many organisations don’t create well-organised libraries of code that can be easily reused or composed into new solutions. This means that developers struggle to find the correct code to use and so decide to create their own. Developers tend to focus on building code for a specific purpose rather than create a generic component that could be reused. In some cases, developers will cut and paste together fragments from Stack Overflow (or other developer community) without really understanding what they do. This can introduce unknown dependencies on external libraries that can open vulnerabilities into the solution.

Data Migration

A large part of the cost and risk of many development projects is the migration of data from old applications and services into the new service. Trying to maintain many years of legacy data can compromise the design of the new code. The new code will have to deal with both the functionality required by the new system plus the overhead of trying to match new and old data models. Error handling will need to be added to compensate for old data that is “incomplete” or “invalid” under the new data model rules.

Why IT operations fails to support the business?


Poor Strategic Alignment

Many organisations suffer from poor communication between the Business and IT organisation. In some cases, poor communication within the IT organisation can also lead to issues with the delivery of IT solutions to the business. A failure to communicate leads to the Business and IT pursuing different objectives which leads to IT spending time and effort on deliverables that don’t directly support the immediate business needs. An effective communications and governance process should help Business and IT to agree what the priorities are and ensure that effort is focused on deliverables that add value to the business.

The “Integration Hairball”

Many organizations have multiple legacy applications and services containing silos of data. Data may be “shared” between applications using batch ETL (Extract, Transform, Load) at various frequencies, or sometimes even real-time through APIs (Application Program Interfaces). Over time the addition of many applications and services increases the complexity of integration, making the overall business solution fragile. The IT team come to fear making changes because of the business risk and cost. Migrating to micro-services isn’t always an easy answer – too frequently we just end up creating even more micro-silos of data. This in turn creates more integrations, reduces application performance, and doesn’t improve the fragility of our overall solution.

Data Analysis and Reporting

Data that is distributed across many applications and services is hard to assemble and integrate in real-time. Challenges to overcome include the underlying security models of the legacy applications and correlation of data between data silos. For example, an “Order” field in one application may be assigned a unique internal numeric id, whereas the “same” order in a different application may be identified by its “Order Number”. For many organisations, reports that span multiple applications must be created by manual processes using tools such as Excel. Larger organisations spend large amounts of time and money creating data warehouses which again increases the fragility of solutions as we have now introduced an extra layer of application data integration.

Single Customer View

Many organisations find it difficult to create a single view of their customer data when information relating to the customer’s services may be stored in several different applications. This leads to a proliferation of “swivel ware” (a user switching between screens in different applications) in organisations – especially those that have grown through a merger and acquisition and IT systems have not been fully integrated. This causes issues with customer service and satisfaction making it difficult or even impossible to handle customer requests seamlessly when they require the use of multiple applications. An obvious consequence of this problem is a dependency on manual workflow to complete simple tasks. This consumes additional time, effort, cost, and risk of introducing bad data.

Poor adoption of standards and processes

Many Small and Medium sized Enterprises (SMEs) do not have formal standards and processes that are consistently used in the production of IT deliverables. Deliverables are created on a “best effort” basis by a series of developers. Frequently, software is re-developed because the new developers didn’t like the way the previous developer wrote the code. Creation of software that isn’t standardised leads to a proliferation of languages, libraries, and frameworks in use within the same organisation. This causes sustainability issues as developers join and leave the team. Not enshrining “best practices” within standards and processes can lead to inefficiencies in software development, unnecessary errors in deployment into production and more fragile IT solutions.

Inflexible Workforce

In organisations that have grown through merger and acquisition it is common to see that the underlying IT solutions that support the business are never fully integrated. One of the reasons for this is that the IT solutions for each part of the business were developed using different languages, libraries and frameworks. This means that the new business now has too many IT solutions with potentially overlapping functionality. Each of the merged organisations had its own team of developers that specialised in their own technology and are unwilling or unable to adapt to the new situation. Moving people between teams as demand fluctuates over time is difficult or impossible which means that some people are overloaded whilst others are idle. This can also cause long delays in getting changes made on one set of applications because of a lack of available resources in the relevant skillset.

Unsustainable Solutions

We have all found ourselves taking ownership of existing business solutions where we end up scratching our heads thinking “why did the previous developers do that?”! As software developers, many of the solutions we see simply aren’t sustainable. Maybe they rely on applications or technologies that are no longer supported. Maybe there are so many application systems dependencies that the risk of making a change is simply too high. Maybe the software is poorly structured or overly complex. This can leave us feeling backed into a corner with no good choices to make the changes that the business is now desperate to deliver.

It’s time to level up!


We have undoubtedly made great strides in our ability to build complex systems using modern DevOps, Agile Project Management, Domain Driven Design and micro-service architecture over the past decade. Although our industries knowledge base is maturing, many organisations are still having major issues with adoption and implementation of modern development practises leaving many IT departments stuck in a rut.

Part of the problem is that we are still trying to solve these issues in fundamentally the same way. Our development projects are still at risk from fuzzy requirements. We are still writing lots of software (creating large liabilities) to provide even basic functionality (minimal assets). Our business solutions are still fragile, complex, and expensive to change. Micro-services are a step in the right direction, but we are still seeing issues in the implementation caused by further fragmenting data between services and trying to find caching and integration approaches to maintain adequate performance.

The Return on Investment (ROI) calculation for our approach to developing software is still fundamentally flawed. To change the ROI equation in our favour we must take a different approach to software development. We need to ask some basic questions about how we build software based on everything we have learnt about software engineering over the past three decades.

  • What if we replaced most of the software with data?
  • What if we created truly re-usable components that we can assemble into new solutions?
  • What if we eliminate all data silos and created Data as a Service?

The idea behind Data as a Service is to systematically reduce the need for Code whilst delivering more Functionality.

What is Data as a Service?

Data
at Rest

At rest within a persistent data storage medium

Data
in Motion

When being shared with third parties or other internal solutions

Data
in Action

In action when we want to perform some business logic on it

Data
Presentation

Presented as part of an omni-channel user experience