A week of IT and graph of the week

A busy week of IT at FlockData,
and your graph of the week for May 26, 2014!

One of the powers of a graph database is the ability to connect things with things and FlockData uses the industry leading open source DB Neo4j. This graph image is a study of cervical cancer studies that reference other studies. We applied a colour scheme based on the date that the study was published. The pinker the dot the more recent the publication. This chart was produced out of PubMed data that we are processing, in FlockData, on behalf of a medical startup. We used Linkurious to produce the visualization. The entire exercise from loading the data to charting the results took 4 minutes. To be honest, it was tricker getting the image into Confluence.

FlockData graph of the week!

FlockData graph of the week! May 26, 2014

Thank goodness for failing tests

Another drama this week was sporadically failing integration tests on an AWS instance that executes our tests. I love tests and FlockData has the usual unit & functional tests and it also has an integration & scenario suite. In all we run 124 tests as at the time of writing. Our integration suite is treated as an application in its own right and we value the feedback it gives us. A failing test is a sign of faulty code, fault assumptions or, in the case of integration, maybe a lag factor. Either way, the failure has to be sorted because it’s telling us something important. Looking at our Design and the component stack involved and you can see that there is a lot of complexity that we shield people from. All this complexity needs to be tested and for us, the icing on the cake is scenario based integration tests that mirror operations in the real world. Code doesn’t get in to our releases until all tests pass.

FlockData data flow

FlockData data flow

While tracking down the integration failure, we wanted to ensure that we could forensically trace what was going wrong where in the services via the logs that were being generated. This will provide output for the future that will assist others with remotely debugging issues. So after getting the tests resolved we have massively improved the output of our logs. Unfortunately that’s not adding functionality, but it’s part of our commitment to quality.

Debugging the issues also revealed a few holes in the way we were externalising the configuration and controlling behaviour via configuration switches. Fixing the tests meant fixing the config issues which we documented in JIRA and the Installation guide. It’s always nice to get the green light don’t you think?

Feedback from many sources is always welcome, both people and systems


Last week a security bulletin was put out that necessitated the patching of this wiki – Confluence. Unfortunately we were running a fairly old version and  database level schema updates had to be applied as well. While documentation is very good, it still took a long time to finish. Upgrade MySQL to supported version, update confluence, upgrade data, backups, restores on and on. If I never have to upgrade Confluence on MySQL again it will be too soon.

All’s well that ends well and we have a shiny new 5.5.1 install of Confluence to integrate with our Bamboo and JIRA deployments.

A little bit of customisation of Confluence and we have a reasonably good looking front page for documentation.

Final thoughts on quality

The service we offer, how we do it, and what we do come together to provide a quality service. Having a solid and transparent knowledge base on hand to capture and collaborate is of paramount importance to me. The Atlassian tool suite links a lot of things across our integrated build environment to ensure that knowledge is retained and that we can learn from the past. We always want to ensure that we have a quality release on hand to make available especially when dealing with complicated environments. What we learn stays with us – let’s work smarter, not harder.

FlockData build process

FlockData build process

Comments are closed, but trackbacks and pingbacks are open.