What are Data Agnostic Services (DAS)?

The essence of the Data Agnostic Services approach is to see data as a generic quantum of information where the content is immaterial. That is, we do not care what type of data is being stored. It could be information about a Car, Customer, Person, Insurance Policy, etc. The storage of the data does not depend on the format of the information. We can achieve this by storing the information in a neutral format such as XML or JSON.

The next step is making the storage of information independent of its usage within an application or service. This means that data should be stored in a single System of Record (SOR) within a centralised location and not be distributed or duplicated within multiple applications or services. All applications and services access the same data repository, which exposes the latest version of the data. This will prevent issues with performance, synchronisation and consistency.

We can then develop a set of services that handle data within four distinct zones:

At rest within a persistent data storage medium
Presented as part of an omni-channel user experience
In action when we perform business logic on it
In motion between data stores (for example when being shared with third parties or other internal solutions)

Separation of Concerns

In the DAS model, a service is a distinct piece of functionality that acts on data. We enforce a strict separation of concerns between the services we create. Each service exists totally within one of the four DAS zones shown above.

So, a service in the “Data Presentation” zone strictly controls the user interface. If data needs to be saved, it calls a “Data at Rest” service to do that. If data needs to be acted on by some form of workflow, it uses a “Data in Action” service to do that.

We never mix functionality within a single service – so there isn’t a case where there is software within a form writing data directly to a database or attaching business logic to the code behind a button.

This allows us to produce simpler services with clearly defined interfaces that hide the internal implementation of their functionality. We can modify how each service works independently of all other services. For example, we could completely replace any of our data storage services without impacting any of our data presentation services that call them.

Data Agnostic Services allow us to reuse our services in multiple solutions. For example, the service used to save a Car is the same as the service used to save a Person or Insurance Policy since the data is stored in a neutral format.

This reduces the complexity of the overall business solution and makes it easier to sustain.

Security Boundaries

As data crosses any of the red lines in the diagram, it is crossing a well-defined security boundary protected by firewalls and other intrusion detection devices. We can also apply technology such as virus and anti-malware software within each zone to prevent the spread of malware or viruses.

As data crosses a zone boundary, we audit the transfer by timestamping it and tagging it with the credentials of the user to validate their access to the data. This lets us track trends and isolate suspicious or anomalous activity.

Our security boundaries allow us to create an in depth defense utilizing different technologies to secure different data services. We can add monitoring, threat detection and alerting within each data zone to initiate a rapid response to any security incident so that we can isolate and shut down impacted services.

Another advantage to the separation of concerns and security boundaries is that most mistakes in legacy code that allow for things like injection attacks are blocked by DAS. For example, even if a service in the Data Presentation zone allows a user to enter an injection attack, it can’t access the data storage directly. It must instead call a Data at Rest service that protects the data storage.

Inherently Sustainable

In an ideal world, we want to leave business solutions better than we found them. We need to focus on creating simple and sustainable software that the next set of developers can evolve and extend as business needs change.

To create a sustainable solution, we start with the development of a sustainable architecture. This architecture needs to be well documented, understood, and enforced by all our team members. If we go outside the architecture because we absolutely need some new technology, then we should extend our architecture to include the new technology in a way that complements and enhances our existing solution components.

As we build new components and services we consider their full life-cycle, which includes how they will ultimately be retired and replaced. We can make it easy for the next set of developers to understand, maintain, or replace any component within the solution without running the risk of bringing the whole solution down.

The DAS approach naturally lends itself to the development of sustainable solutions. By maintaining a strong separation of concerns and well-defined security boundaries, we can create small services with well-defined interfaces that hides the inner workings of the service.

By dividing the Data at Rest into its own services we are minimizing the issues normally associated with data silo creation within applications and services. This means that any service can access the latest version of any data it needs - if the user has appropriate permissions.

Services in the other zones (Data in Action, Data in Motion, and Data Presentation) can be smaller in scope since the security and synchronisation is dealt with inside the Data at Rest service. For example, if we need to add new functionality to an existing solution we can encapsulate that functionality into it own separately versioned and deployable service. This new service would be totally isolated from existing services and would be compartmentalised within its own security boundary.

Component Library

A key objective of the Data Agnostic Services approach is to break the link between functionality and code. Ideally, we want to be able to create and deploy functionality without writing any code. To achieve this, we want to make sure that when we are forced to create code we maximise its reusability.

The first step is to break down the required functionality into a set of reusable functions. This doesn’t mean writing functions that are generalised and offer many options through configuration or parameters, which leads to overly complex code. Trying to address possibly unnecessary future requirements wastes time and effort. We want to focus attention on the problem that needs to be solved now. We might need to break a bigger problem down into small and reusable functional building blocks. We then code the building blocks from which we can implement the required functionality with a view to being able to reuse or rewrite those blocks later.

For example, consider a simple requirement to send an email to a user. The general problem is outbound communication – we could build a general service that can send emails, SMS, push notifications, etc. At this point that general functionality is not required - we just need to be able to send an email! We don’t need to invest the extra time, effort and complexity to build something that we may never use.

The most efficient approach is to start by building a very simple function that just sends an email and integrate that into our solution. Job done – minimum cost and time to develop a simple and reusable building block that will allow any of our solution components to send an email.

At some future time, we may need to send an SMS, then we will create a building block that just sends an SMS and integrate that into our code.

Later, we may need to send a message to a person either by email or SMS depending upon preference. We will create a new building block that will check which is the preferred channel for the user, and use either the email or SMS building block to send the message.

Small and simple building blocks should be our preferred development approach as it enables us to build the absolute minimum amount of code to implement the required functionality quickly. Designing for reuse or rewriting means that we can later combine them in new ways to implement more complex functionality without breaking or re-implementing earlier solutions.

For these building blocks to be fully leveraged we have a Component Library where our developers can easily find them. The Component Library holds all versions of all building blocks with clearly identified stable and developmental builds. It's easy to search for existing building blocks that provide functionality that has already been developed, tested and proven in production. Building blocks are stored along with their associated documentation to make it easy for developers to understand their purpose, interfaces and implementation. That way they can be reused as is or enhanced to provide additional functionality. At the project management level, it is a requirement that developers first search for and reuse an existing building block before creating their own.

What are Data Agnostic Services?

Data
at Rest

within a persistent data storage medium

Data
Presentation

as part of an omni-channel user experience

Data
in Action

when we perform business logic on it

Data
in Motion

between data stores