When I think back to my humble roots in web development, there was only production. Editing files directly using an FTP client, or SSHing into a server to fix a bug - they were some cavalier moments.
Things have moved on a lot since then. Tools have been created, processes have been developed and we've now got this amazing thing called <a href="https://en.wikipedia.org/wiki/Continuous_delivery">continuous delivery</a>.
Some things haven't changed though. So many times I've found myself either in public or private projects where an API will only work in production. The strangest edge case you've ever heard of will only pop up in production. Your product manager will insist upon seeing a feature in production, sometimes right before canning it entirely.
And when we find ourselves in similar situations, we cover our eyes with our hand, peek through the gaps and press the big red "Ship it!" button, even though we are taught (usually through experience or by others who have had the misfortune) to never do this. Especially on a Friday.
You'll get it in the neck. You'll be up until 3am fixing it. You'll have to bug your tech-lead who is on holiday for a script to fix it. Yadda yadda yadda.
I have been on both ends of this (the fixer and the fixee) and what I can confidentally say, more confidentally than I know the sun will rise in the morning, that I will be on both ends again. 
<blockquote>
Happiness can only exist in acceptance - George Orwell.
</blockquote>
I think the reason I'm so accepting of this is either that I have ripped the band aid off so many times the pain has made me numb, or I've tried to make testing in production relaxing experience - dare I say - a joy.
<h2>Feature flags</h2>
Feature flags (or toggles) are truly amazing. If I'm being honest I wanted an excuse to write an article about feature flags and I've not-so covertly managed to sneak it into an article about testing in production.
Why are they so good? They allow you to "wrap" different sized parts of your application in conditions which can be dynamically turned on and off. There's a few platforms which handle the "server" part of this for you, such as <a href="https://unleash.github.io">Unleash</a> and <a href="https://launchdarkly.com">Launch Darkly</a> - you store the toggles and logic on these platforms, and they afford you different "strategies" in which these can be activated and an API (or SDK) to access them.
The simplest strategies are a binary on-off, but you can also have user-based strategies which can check a whitelists, ranging to gradual roll-outs (for example, a percentage of your visitors) right through to custom strategies.
This really shines when you've got a nice single page app which is driven by an API. So long as the feature you're wrapping doesn't inadvertantly break something else on the page (GraphQL - I'm looking at you) and you have set a sufficient strategy that means random visitors are not going to see the feature, you can pretty much test whatever you like in production.
This does not come without it's drawbacks though. There's a fair bit of investment to make - in terms of time for building these wrappers into your UI (and maybe building a proxy endpoint for grabbing the feature flags in your API), and also financially in that you will need somewhere to host the feature flag service (either yourself or paying for a platform). You can achieve a very basic and crude implementation of feature flags with environment variables so that's always an option if you want to give this a try.
Also there's the opportunity for a less-experienced developer to open massive security holes with this too. Oh, I checked the Network tab and saw that <code>newSuperSecretButUnsecureLoginPage</code> is <code>false</code>? What if I set that to <code>true</code>?
With feature flags though you get, which I think is their most criminally underrated feature, is the ability for other people to deploy your features for you. There's been a handful of times were I've had to stay late (or rise early) to push something out, usually sat around awaiting the "go-ahead". It wouldn't be so bad, but you've still got to actually get through the deployment ritual and it's usually when something in CD decides to change or break. Oops, didn't add that environment variable. Oh no, now a test broke. And now I've fixed it coverage has decreased. It's a spiral of doom, why would you not want to instead just flick a (virtual) switch?
<h2>Logging</h2>
Feature flags are great on the front-end (and there's nothing to stop you using them elsewhere) but for back-ends it's hard to be as "visual" with your testing. What do you actually need to surface for testing?
Logs are your friend. I will be honest and say that it wasn't until I transitioned from web development into software development that I really used logs. Yeah there's the nginx logs or whatever - great, looks like I'm getting hacked. But actual application logging is super powerful and when used correctly allows you to build a very visual narrative to how your application is behaving.
Invest time in learning about your logger. Winston, the popular Node.js logger, is what we use day-to-day and a colleague of mine pointed out that you can create a parent logger and give it some attributes. This means you can search logs on an attribute and get all the subsequent logs back.
For example your application might look like this:
<pre><code class="language-javascript">import logger from 'winston';

function updateUser(userId) {
 logger.info(`Trying to update user ID: ${userId}`);

 try {
 // ...
 logger.info(`Successfully updated user ID: ${userId}`); 
 } catch (error) {
 // ...
 logger.info(`Failed to update user ID: ${userId} error: ${error.message}`); 
 }
}
</code></pre>
and your log lines will look not dissimilar to:
<pre><code>Trying to update user ID: 123
Successfully updated user ID: 123
Trying to update user ID: 456
Successfully updated user ID: 456
Trying to update user ID: 789
Failed to update user ID: 789 error: Network request failed...
</code></pre>
Not bad, but not great either. If we want to find out everything that happened to user <code>789</code> we have to do a fuzzy search which could throw up other things containing those numbers. However if we refactor to:
<pre><code class="language-javascript">import loggerFactory from 'winston';

function updateUser(userId) {
 const logger = loggerFactory.child({ userId });
 logger.info(`Trying to update user`);

 try {
 // ...
 logger.info(`Successfully updated user`); 
 } catch (error) {
 // ...
 logger.info(`Failed to update user`, {
 errorMessage: error.message,
 }); 
 }
}
</code></pre>
Our log lines now look like:
<pre><code>Trying to update user {"jsonPayload.userId": 123}
Successfully updated user {"jsonPayload.userId": 123}
Trying to update user {"jsonPayload.userId": 456}
Successfully updated user {"jsonPayload.userId": 456}
Trying to update user {"jsonPayload.userId": 789}
Failed to update user {"jsonPayload.userId": 789, "errorMessage": "Network request failed..."}
</code></pre>
Much better! We can now do a search like <code>jsonPayload.userId=789</code> and get the full story.
Hopefully you didn't need to read this article to realise logging is ace. It doesn't come without it's pitfalls though. If you trawl the logs of most tech companies you're bound to find some kind of Personally Identifiable Information (PII) floating around in there. So be careful.
<h2>Dry mode</h2>
If you're using logs to their potential then you can begin to use them to tell a story. Front-end work is, by it's visual nature, straightforward to demo to stakeholders, QAs or other engineers, but what do you do if you're a back-end engineer?
One thing I've tried to push when doing back-end work is being able to run things in some kind of "dry" mode (also called "shadow testing" or "toothless mode", etc.). When running in this mode a task would be carried out as it normally is, with lots of logging, however any calls to mutate any data are wrapped in a "are we running in dry mode?" check. These then just log what they would do (rather than actually doing it) and continue on.
Sounds simple I know but I love dry mode. I've had to deal with some extremely fiddly issues with a event-based distributed system and guess what... the data fetching only worked in production. We'd invested a lot of time creating an exact replica of the environment and thoroughly tested our code with several QAs, only to find several transient issues when ran in dry mode in production.
Again we have limitations as it's difficult to test any data which depends on any data that would have been mutated. You can use things like database factories or dependency injection to use fakes rather than the real rows but then you've got to justify this extra time investment for code you're probably going to delete at some point in the near future.
<h2>Conclusion</h2>
It's about trade-offs, and common sense too. I may have painted a rosy picture about testing in production but would you do this if it were avoidable? Hell no. Invest your time in things like building proper CI/CD pipelines, blue-green deploys which allow you to hit that deploy button at 5pm on a Friday, beer in hand.

Testing in Production

Although I don't get paid to write PHP anymore (my full-time job is full-stack JavaScript/Node), I still like to dabble with <a href="https://laravel.com">Laravel</a> as it's "batteries included" approach makes it really easy to knock together a concept in a matter of hours.
Something we've been rolling out at work at the moment is the integration of <a href="https://unleash.github.io">Unleash</a> into some of our projects. If you are unfamiliar with Unleash, it's an open-source feature toggle package which aims to be trivial to deploy, yet fully featured with a plethora of strategies to choose from - you can even make your own.
If you are unfamiliar with feature toggles/flags (spoiler: they are awesome), I'd suggest reading up on them otherwise this package will not really make that much sense! A TL;DR on feature toggles/flags:
Hiding features in your codebase behind toggles that are controlled externally.
Some of the benefits of this are:
<ul>
<li>You can experiment with rolling these features out, great for measuring the impact of a particular feature.</li>
<li>Non-technical people can choose when to roll out new features, for instance to coincide with a marketing push, taking the burden away from the developer.</li>
<li>You can test things in production without having to worry about degrading the experience for actual users.</li>
</ul>
Doing a bit of searching on <a href="https://packagist.org">Packagist</a>, there didn't seem to be anything that was compatible with Laravel which I thought was a bit of a shame as Laravel's <a href="https://laravel.com/docs/blade">Blade</a> syntax would lend itself perfectly to working with feature toggles.
So here's my attempt at an Unleash client:
<a href="https://github.com/mikefrancis/laravel-unleash">https://github.com/mikefrancis/laravel-unleash</a>
It's still very much a work in progress as I am still building strategies to handle all the default Unleash strategies.
If anyone would like to give it a whirl, I'm looking for testers at the moment to see how stable it and if any performance improvements can be made.
Happy feature toggling!

aditya-sahoo-iZxInmFP02U-unsplash

Laravel Unleash client

I'm a huge fan of <a href="https://nextjs.org">Next.js</a>, it's my go-to framework if I'm building a web application these days - I tend to leave Gatsby for blogs and heavily content-driven sites, and create-react-app for more single page-y
applications. When v9 dropped the other day, I was reading through the <a href="https://nextjs.org/blog/next-9">blog post</a> and immediately wanted to see if I could hack something together using the new API routes feature.
If you're not familiar with Next.js one of it's advantages over other frameworks is the minimal approach to configuration. All you need is a <code>./pages/index.js</code> file, containing:
<pre><code class="language-javascript">export default () => &#x3C;h1>Hello world!&#x3C;/h1>;
</code></pre>
and you have a working, server-rendered React page for your Next.js app. Version 9 brings API routes, meaning all you need is a <code>./pages/api/index.js</code> file, containing:
<pre><code class="language-javascript">export default (req, res) => res.end("Hello world");
</code></pre>
and you have a working API endpoint. It was possible to do this in previous versions of Next.js by editing the underlying
express server, but having applied the ethos from the rest of the framework, this is available with zero configuration.
<h2>What if...</h2>
This got me thinking, what if you could use the API routing to expose a GraphQL API? Especially with the advent of the serverless paradigm and the return of the monolith, it's becoming increasing
normal to bundle your API endpoints with the rest of your application. So I set myself the task of exposing a small GraphQL API from some Next.js API routing.
If you want to skip to the end, here's the finished repo: <a href="https://github.com/mikefrancis/nextjs-9-graphql-api">https://github.com/mikefrancis/nextjs-9-graphql-api</a> and here's a demo: <a href="https://next-9-graphql-api.mikefrancis.now.sh">https://next-9-graphql-api.mikefrancis.now.sh</a>
<h2>Development</h2>
Start off by creating a new Next.js application:
<pre><code class="language-bash">mkdir next-graphql-api
cd next-graphql-api
</code></pre>
Then initialise a new <code>yarn</code> project (you can use <code>npm</code>, I use <code>yarn</code> 🤷‍♂️):
<pre><code class="language-bash">yarn init -y
</code></pre>
Then install the dependencies needed for Next.js, plus <code>axios</code> for HTTP requests and <code>graphql</code> for parsing and creating GraphQL schemas:
<pre><code class="language-bash">yarn add next react react-dom axios graphql
</code></pre>
Create an test index page - <code>./pages/index.js</code>:
<pre><code class="language-javascript">import React from "react";

export default () => &#x3C;h1>Hello world!&#x3C;/h1>;
</code></pre>
Then boot the server by running <code>yarn next dev</code> and visit <a href="http://localhost:3000">http://localhost:3000</a> to check everything went smoothly. You should see <code>Hello world!</code>.
<h3>GraphQL</h3>
Now we want to create our GraphQL endpoint. Let's create <code>./pages/api/graphql.js</code> with the following content, which we will step through line-by-line:
<pre><code class="language-javascript">import { graphql, buildSchema } from "graphql";

const schema = buildSchema(`
 type Query {
 hello: String
 }
`);

const root = { hello: () => "Hello world!" };

export default async (req, res) => {
 const query = req.body.query;
 const response = await graphql(schema, query, root);

 return res.end(JSON.stringify(response));
};
</code></pre>
Firstly by creating the file with the name <code>graphql.js</code> Next.js will create an endpoint with the URL <a href="http://localhost:3000/api/graphql">http://localhost:3000/api/graphql</a>. Ace!
Then we import the necessary functions needed from the <code>graphql</code> library.
<code>buildSchema</code> allows us to define (using GraphQL - the query language itself) through types what kind of queries we can allow to be requested - or a "schema". In this case, we create a query with
the name <code>hello</code> that will respond with a <code>string</code>.
We then need to define a resolver for this query, always think about these in pairs - here's my query and here's what will resolve my query. These are objects where the properties are the names of
the queries we defined in our schema, and the values are functions which will return the data. Our query <code>hello</code> should return <code>string</code>, so we return <code>Hello world!</code>, but this could be any string -
feel free to try out changing this!
Once we have our GraphQL bootstrapping done, all is left is to create (and export) the function which will serve our requests to <a href="http://localhost:3000/api/graphql">http://localhost:3000/api/graphql</a>.
If you've ever used the express (or koa) Node.js HTTP frameworks, this function signature should look fairly familiar. We are given access to the request (<code>req</code>) and response (<code>res</code>), so we want to take the
supplied GraphQL query out of the request body, then pass this (along with the <code>schema</code> and <code>root</code> resolver) to the <code>graphql</code> function, to resolve our query.
If successful, the response is returned as a plain JavaScript object. We then need to convert this into JSON if we are to send it back to the client (be it a browser, mobile application, Postman etc).
So now, if you hit the endpoint in Postman (I am using <a href="http://insomnia.rest">Insomnia</a> below), you should receive a response. Hooray!
<img src="//images.ctfassets.net/w1rfv8jievsk/5M3Q41EnpNtwv8SLrqbbgA/816f1284259e2600b67570a06f6eff2a/Screenshot_2019-07-12_at_13.03.52.png" alt="Example response">
<h3>Consuming this data</h3>
Back to <code>./pages/index.js</code>, we want to use our newly created GraphQL endpoint to populate our UI. Add the following and again we will go through line-by-line:
<pre><code class="language-javascript">import React from "react";
import axios from "axios";

const query = `{ hello }`;

const App = props => &#x3C;h1>The response from the server is: {props.hello}&#x3C;/h1>;

App.getInitialProps = async () => {
 const response = await axios.post(`http://localhost:3000/api/graphql`, {
 query
 });

 return { ...response.data.data };
};

export default App;
</code></pre>
This is a fairly standard React Function Component. The juicy bit is when we hook into Next.js' <code>getInitialProps</code> method. This is fired by the main Next.js application component, which wraps all
components in our application. It's great for things like data fetching, where you want to do some pre-fetching before a component is rendered or indeed given props. We need this!
We hook into this and use the <code>axios</code> library, a wrapper around <code>fetch</code>, to request our newly created GraphQL endpoint. We supply the raw GraphQL as the body using the <code>query</code> field, then once we
get a response, destructure that into the props which this method will return - so anything that is returned will end up as a <code>prop</code> in the main component. We don't deal with errors or anything like
that, so it's something to think about if you were to take this further.
Then on the lines above we know, having used <code>getInitialProps</code> to supply this, that we will have a <code>prop</code> named <code>hello</code>, as this is returned from the GraphQL query, with a value of <code>Hello world!</code>.
<h2>Conclusion</h2>
Next.js is an excellent framework which allows you to get up and running very quickly, with minimal configuration needed. Creating UI and API routes are trivial, and as we are in full control of
what response is returned from API routes, there's nothing to stop us creating a GraphQL API which sits inside the rest of our application.

Software Engineer & UI Designer based in London, UK. Currently building and scaling a modern, renewable energy provider at Bulb.

Latest Posts

Testing in Production

Laravel Unleash client

Building a tiny GraphQL API using Next.js