Service healthcheck pattern

This is a simple pattern I’ve used on a number of projects to implement healthchecks as part of service monitoring. Whether you are making a scaled complex microservice solution or a simple web to API to database solution, having a set of consistent healthcheck URLs for all your components is extremely useful.

It requires two URLs on each web component in your architecture:

  • Healthcheck (e.g. /healthcheck) – returns 200 if the component thinks it is working correctly and can use it’s dependencies. This means checking it can hit any datastore required, reach any APIs it uses.
  • Status (e.g. /status) – returns 200 with no side effects

If either endpoint returns something other than 200 you know something is wrong, the healthcheck URL may respond with details on what’s failed but the status URL simply responds if the service is reachable and alive.

Most documentation on healthchecks only concern themselves with a healthcheck URL, which is fine when you are dealing with an individual component, but when you have a number of interconnected components which may also be interrogated by other applications (service discovery, load balancer etc.) having a status endpoint which does not trigger side effects is important.

Healthchecks can be expensive, making IO/network calls which place load on not just the component but other resources and services. A common issue is healthchecks causing cascades of requests to dependencies and other healthchecks after a release, which may cause false positives for failure when services take longer than usual to respond. Having the status URL means you can reduce the number of calls to the expensive healthcheck, offloading calls that only need to know if the component is alive.

Advertisements

Paging and sorting pattern for non-Javascript and Datatables

I like JQuery Datatables. It’s a easy to use JQuery plug-in that allows you to enhance an HTML table to support paging/sorting/filtering and all sorts of functionality with little configuration. It supports server side processing (something I’ve blogged on before) to allow serving large datasets.

But it has some issues.

By default it isn’t responsive and doesn’t play nice with small screens, it’s hard to style if you are using custom styling for your website and it will cause some accessibility issues. Also it needs Javascript, so sites that need to support no-js can’t rely on it for paging large tables.

Since I like the Datatables command patterns for API calls and don’t like duplicating logic, I’ve created this sample projectwhich shows how you can implement your model/view/controller logic and back-end logic to support serving a paged/sorted table both with Datatables and pure HTML GET requests on a page. This cuts down on the amount of logic needed and provides an easy to follow pattern for retrieving and using the paged data.

Even if you do not want to use Datatables it’s always good to use an approach which will be familiar to other developers and have a pattern that encourages code reuse and consistency.

Here’s the sample Person list using Datatables:

person-list-js

Here’s the same page with Javascript disabled using HTML GET requests for paging/sorting:

person-list

Datatables provides the quick AJAX redraw of the table with enhanced paging/sorting functions, while the HTML GET provides the non-Javascript support.

To implement this I used a number of classes with generics/abstract methods to allow re-use for different pages/tables:

PagedSortedViewModel – model that can be used for both JSON serialization in Datatable server-side and rendering HTML table.

   public class PagedSortedViewModel<TData> : IPagedSortedViewModel
   {
       public int Draw { get; set; }
       ...
       public IEnumerable<TData> Data { get; set; }
       ...
   }

PersonPagedSortedTableController – controller with routes for both HTML GET and Datatables JSON call

    public class PersonPagedSortedTableController : Controller
    {
        ...
        [HttpGet]
        public async Task<IActionResult> Index(int start = 0, int length = 10, string orderColumn = "Name", bool orderAscending = true)
        {
            var model = await GetPagedSortedResultsAsViewModel(0, start, length, orderColumn, orderAscending);

            return View(model);
        }

        [HttpGet]
        public async Task<JsonResult> DatatableJson(int draw = 0, int start = 0, int length = 10)
        {
            var isAscending = Request.Query["order[0][dir]"] == "asc";
            int columnIdentifier = Convert.ToInt32(Request.Query["order[0][column]"]);
            string orderColumnName = GetColumnName(columnIdentifier);

            var model = await GetPagedSortedResultsAsViewModel(draw, start, length, orderColumnName, isAscending);

            return Json(model);
        }

        private async Task<PagedSortedViewModel<PersonResultItem>> GetPagedSortedResultsAsViewModel(int draw, int start, int length, string orderColumn, bool orderAscending)
        {
            var result = await _pagedSortedRepository.GetPagedSortedResults(start, length, orderColumn, orderAscending);

            return new PagedSortedViewModel<PersonResultItem>
            {
                Draw = draw,
                ...
                Data = result.data,
            };
        }

        private string GetColumnName(int columnIdentifier)
        {
            switch (columnIdentifier)
            {
                case 0: return "Name";
                ...
            }

        }
    }

AbstractPagedSortedRepository – abstract repository class that has a number of virtual and abstract methods, wiring together the queries needed to return the paged/sorted result set so that minimal custom logic is needed for each different table.

    public abstract class AbstractPagedSortedRepository<TResultItem> : IPagedSortedRepository<TResultItem>
    {
        public async Task<PagedSortedResult<TResultItem>> GetPagedSortedResults(int start, int length, string orderColumn, bool orderAscending)
        {
            var innerJoinQuery = GetQuery();

            var recordsTotal = await GetRecordsTotalQuery(innerJoinQuery).CountAsync();

            var whereQuery = GetWhereQuery(innerJoinQuery);

            var recordsFiltered = await GetRecordsFilteredQuery(whereQuery).CountAsync();

            var sortedWhereQuery = GetSortedWhereQuery(whereQuery, orderColumn, orderAscending);

            var pagedSortedWhereQuery = sortedWhereQuery.Skip(start).Take(length);

            var data = await pagedSortedWhereQuery.ToListAsync();

            return new PagedSortedResult<TResultItem>
            {
                recordsTotal = recordsTotal,
                recordsFiltered = recordsFiltered,
                data = data,
            };
        }
        ...
    }

PersonPagedSortedRepository – Implementation of the abstract repository for a table showing joined results of the Person/Party entities.

public class PersonPagedSortedRepository : AbstractPagedSortedRepository<PersonResultItem>
    {
        ...
        protected override IQueryable<PersonResultItem> GetQuery()
        {
            return from p in _partyDbContext.Parties
                   join o in _partyDbContext.Persons on p.PartyId equals o.PartyId
                   select new PersonResultItem { PartyId = p.PartyId, Name = p.Name, EmailAddress = o.EmailAddress, DateOfBirth = o.DateOfBirth, DateCreated = p.DateCreated };
        }

        protected override IQueryable<PersonResultItem> GetSortedWhereQuery(IQueryable<PersonResultItem> whereQuery, string orderColumn, bool orderAscending)
        {
            switch (orderColumn)
            {
                case "Name": return orderAscending ? whereQuery.OrderBy(x => x.Name) : whereQuery.OrderByDescending(x => x.Name);
                ...
                default: return whereQuery;
            }
        }
    }

The view renders the table, and has Javascript to use Datatables if Javascript is enabled (hiding HTML paging/sorting controls).

Links:

The symmetrical architecture trap

Often when thinking about topics to write about I hesitate, as in retrospect what I’m saying seems obvious. But it’s very common to fall into simple patterns when you are in the thick of a project, doing something which the flaws only become apparent later when early chaos is over and you can think clearly.

One of these traps is symmetrical application architecture, making two similar components in your solution use the same architecture, even when they have different requirements.

A common example of this is when a web application will have a public facing external site and a more secure internal site for administration. On the surface these two components have similarities, they both serve HTML and need to access/persist data, so you may initially use the same architecture for them.

Symmetric architecture diagram

However, you soon realise the external site needs to handle much more traffic than the internal, and it’s data requirements are different (higher read or write, only needing access to specific data). You can resolve this by scaling the architecture, but it’s clunky.

Symmetric architecture with external load diagram

Then you realise the internal site has more security and auditing requirements. You can resolve with implementation changes but it would be neater to include additional layers or services.

Symmetric architecture with internal audit security diagram

The symmetry of the architecture becomes a conceptual barrier to change, changing either one appears to be introducing more complexity but in reality the implementations are diverging anyway due to their different circumstances. Looking at them individually and at how they will be hosted on less abstract infrastructure diagrams can help. Could be your external and internal sites don’t need the same data store or layers, and changing them could save resources and simplify implementation.

Asymmetric architecture diagram

Embracing asymmetry in your architecture early can help you break out of this mindset and prevent you hitting problems later when your implementation work arounds start to creak.

Building and tagging container images in CI

docker-ci-tagging-anon

Been thinking a lot recently about how to manage versioning and deployment using Docker for a small scale containerised solution. It’s different from a traditional release pipeline as the build artifacts are the container images with the latest code and configuration, instead of the CI having a zip of the built application.

In a completely ideal containerised microservice solution all containers are loosely coupled and can be tested and built independently. Their CI configuration can be kept independent as well, with the CI and testing setup for the entire orchestrated solution taking the latest safe versions of the containers and performing integration/smoke tests against test/staging environments.

If your solution is smaller scale and the containers linked together, this is my proposed setup.

Build

Images should be built consistently, so dependencies should be resolved and fixed at point of build. This is done for node with npm shrinkwrap which generates a file fixing the npm install to specific dependency versions. This should be done as part of development each time package.json is updated, to ensure all developers as well as images use the exact same versions of packages.

On each commit to develop the image is built and tagged twice, once with “develop” to tag it as the latest version for develop branch code, and then with the version number in the git repo VERSION.md (“1.0.1”). You cannot currently build with multiple tags, but building images with same content/instructions does not duplicate image storage due to Docker image layers.

Tagging

The “develop” tagged image is used as the latest current version of the image to be deployed as part of automated builds to the Development environment, in the develop branch docker-compose.yml all referenced images will use that tag.

The version number tagged image, “1.0.1”, is used as a historic fixed version for traceability, so for specific releases the tagged master docker-compose.yml will reference specific versioned images. This means we have a store of built versioned images which can be deployed on demand to re-create and trace issues.

On each significant release, the latest version image will be pushed to the image repository with the tag “latest” (corresponding to the code in the master branch).

Design for Devs – Change sequence diagrams

I’ve been asked a few times by junior developers how to get started in designing code, as if it’s some sort of art technique. In truth every developer is doing design, no one spontaneously writes complex formal logic. Most just do it in their head based on experience and patterns they’ve used before. For small well understood problems this is fine, but when you are dealing with larger or more complex changes doing a little design work up-front can really help clarify things and ultimately save time while giving a better solution.

I’m writing this to document a simple type of design I’ve used on many projects, a change sequence diagram, one you can do quickly on paper or on a whiteboard in ten minutes and I’ve found to be very helpful in thinking about what changes are required, the size of the overall change and promoting re-use of existing code.

Here’s an example:

change-sequence-diagram
It’s a pretty simple variation of a sequence diagram, where you show the sequence of events which should occur as a series of interactions between the involved people/components. It normally starts with person, like someone clicking a link on web page, then shows how the system responds. The change part is about highlighting what components in each part of the system need to change to handle the functionality, what parts need to be added/updated/removed.

Doing this forces you to think up-front about what you will need to change and how the system will interact to get the end result. It ties the individual component changes to the overall user requirement, e.g. you’re not just adding a new database column and view field, you’re adding them so the user can see and update their middle name on the personal details screen. This helps you understand how the parts of your system interact and use consistent implementations and design patterns in your changes, plus identify the unit tests and test scenarios.

When you are done, the number and type of changes shows the scale of the overall change, useful for estimates, and breaks it down into manageable chunks of work. You’ll get the best results if you do it paired with someone or get someone else to review your design. Doing this checks that you aren’t breaking existing patterns in the code or missing something that could increase or decrease the complexity. You can expand it to include alternate flows and consider NFR’s for security and performance.

Next time you’re looking at a new requirement or user story give this a try, you’ll be surprised how easy it is to do and what you’ll get out of it.

The incremental complexity trap

This is a common problem which developers face, particularly in agile projects when it’s normal to make user stories related to previous user stories to add new functions to existing screens and flows.

It happens all the time. You start with a something like a simple screen and do the necessary work to make it. Then you get a few new requirements, add some more fields and new validation for alternate user flow etc. You do the same, add the fields to your screens and logic to the controllers. Then it happens again, then again and again. More user flows are added, more fields appearing in some of those flows, more complex validation. Your original simple screen, controller and validation logic is now a monster, unmaintainable and a nightmare to debug.

Often teams don’t do anything to solve it, just live with the problems. Commonly by the time they realise what’s happening it’s easier just to keep layering on the complexity rather than deal with it properly. No one wants to be holding the bag when it’s time to stop and refactor everything.

This can be particularly bad if it’s happening in multiple places at the same time during a project. The effects of shortsighted decisions start to snowball, affecting development speed and increasing regression issues, and it’s not feasible to refactor everything (try selling a sprint full of that to a Product Owner).

So how do you avoid the trap?

  • Keep it simple

Establish code patterns that encourage separation of concerns, make the team aware of them and how to repeat the patterns.

  • Unit tests

Test complexity helps highlight when things are getting out of hand before it’s too late and make it much easier to refactor by reducing the chance of regression issues.

  • Anticipate it before it happens and design

This is the job of the Technical Architect, to know what requirements and stories are coming and how they will be implemented. If an area is going to get a lot of complexity it needs to be handled or you will end up in the trap.

What can you do if you are already in it?

  • Stop making it worse

For complex classes stop adding new layers of logic. Obvious, but the temptation will be there. Stop and plan a better approach, as the sooner you do this the easier it will be.

  • Don’t try for perfection and refactor everything

Big bang refactors are risky and time consuming. Take part of the complexity and split it out, establishing a pattern to remove more.

  • Focus on your goals

Overly complex classes aren’t bad out of principle. They are bad because they slow development and give bugs places to hide. Refactoring and creating a framework to add new functionality can speed development and reduce occurance of defects, which also eat time. You should focus your effort on refactoring areas which will need more complexity later, investing time now to save more later.

Microservice authentication and authorisation

Below is a sequence diagram describing a common pattern for handling authentication and authorisation in a microservice solution, where you have a number of small web services interconnected and don’t want them all having to implement  authentication and authorisation individually, just to be simple services focused on their job.

Microservice authentication and authorisation seq

Using this pattern you can have any number of business microservices, all using the authentication automatically and calling the same authorisation service when necessary, making them simpler to implement and re-using the services easier.

You can go a step further and put the authorisation checks in the reverse proxy against each service endpoint but this requires the reverse proxy to know the actions/permissions of each service, plus it’s likely your services permissions will be dynamic based on execution.

Nginx with lua scripts are a good way of implementing the smart reverse proxy, below are some links about this.