Feature toggling as a design tool?

A tale from dev life

Not so long time ago, I was implementing conceptually simple, but integration-wise challenging piece of functionality.

It was technical capability, cross-cutting concern one, meaning that it might have had big impact on some initialization code for each module.

But back to the point.

I was happily developing it locally, everything was working fine according to the verification done by manual tests (why "verification"? Please check The "ambiguity" of TDD blog post).

In a sense it looked promising, so the next step was actually to verify the functionality in the dev environment.

It was pretty calm, stable time of this component so no need to worry - PR merged, got deployed to dev environment.

How to get free stress points

Ooops, it wasn't working there. Unexpected (or rather unnoticed) issue of Newtonsoft.Json's multiple versions appeared.

"Soon to be fixed", I thought.

I continued working, thinking how to beat that well-known problem.

Suddenly, a support ticket came - one of the customers started facing some issues with a new functionality released some time ago. Initially, it looked like a unexplored case.

"Soon to be fixed", I thought again, but now with different subject in mind.

But, but - wait?! My unfinished, broken functionality is merged, meaning that according to our process, it would be released too.

A glitch in the Matrix. Impossible state became possible?

Is a broken, hidden code a bug or not?

Quite high prio investigation with possible fix to be delivered as soon as possible - that puts some frame around thinking and constrains work in a surprisingly creative way.

There were multiple models for fixing it (a solution to the given problem): reverting my work, working directly with master branch, creating a copy of existing branch, etc.

One of the options was to hide broken code. Literally, make it unavailable.

Best of both worlds: being able to continue working on new functionality and fixing a bug. Isn't that neat?

You probably recognize this as a feature toggle, when one, in the runtime, can change the behavior of the system. But, is it that easy?

Is my design ready to be toggled?

Turned out that I had multiple places to touch so that the broken functionality gets entirely hidden. Can you feel it? It is a bit smelly.

I was missing a proper abstraction (I can hear you - where's my specification? "With TDD you won't reach such state!", but that's another tale).

In the reasonable amount of time the proper abstraction was provided and suddenly I had only two guys (why "guys"? Please check Organization-Driven Design), responsible for acting upon a toggling this feature. Now we are talking!

Soon I fixed the issue, it got delivered and, almost like a state machine, I returned to the initial state, working on the new functionality.

Could this short, but still, stressful situation, be mitigated? Of course!

Possibly, there were various measures I could have taken into account, but actually NOT designing it in the proper way highlighted some interesting properties of this codebase (and other, similar ones?).

As Kevlin Henney said in one of his talks (I believe it was during #FAIL):

(...) we are forced to learn how things are working when they fail.

How easy it is to hide your code?

What would you do in a similar situation? Would it be easy for you to apply the similar approach?

I know, typically we don't need to make some parts of our codebase hideable, because they are either too small or it would require additional effort to make them gaining this trait (e.g. no feature toggling infrastructure, convincing others, etc.).

There are certain actions that probably won't benefit from making them hideable, e.g. adding a property/field to a DTO, working on new method in a class.

My introductory, short tale revealed some weaknesses of the codebases that are difficult for making them "feature toggleable". In the mentioned case, it was a lack of proper abstraction.

What does it mean that a feature is "difficult to hide"?

What properties are missing so that, when "added", code becomes easy to be hideable?

"Hideability" opens next level

A piece of functionality, easy to be hidden, brings another interesting perspective. When we can easily hide it, we should be able to easily replace it, isn't it?

The first thing that comes to my mind is versionability, meaning that we are able to use either one version or another, for respective capability.

We know how to version our APIs. There are various stategies for achieving so. Consumers are not interested how we are arriving at this solution, they need to know the contract.

What happens under the hood? Is our code really able to be versionable?

Example time: say hi to Users.Backend!

Let's imagine we have a very simple backend that gets more and more attention from various customers around the globe.

I bet you know this style of architecture quite well. I would even say it's quite common.

Going to the details of each layer, one can see:

Impressive, isn't it? But again, back to the point.

Imagine that we were asked to introduce a novel change for assigning groups to the users.

As this is our core and critical capability, we need to make sure it's always stable, always working.

Let's highlight everything related to Group concept:

The birth of versioning

That seems easy. We should use a standard way to version the API. Our consumers will get a chance to decide either to use old way or a novel way.

What happens under the hood though?

We simply add a next controller action (at least in ASP.NET Core world) with a proper version number.

We could also introduce a separation between V1 and V2 controllers to retain even more granular control.

Let's add another constraint.

Imagine that this change (because of being novel) requires touching every other related collaborator: GroupsService, GroupsRepository, Group (and maybe something from Users.Backend.Common). By touching I mean either extending something which is already working or modifying it in some sense.

It starts to be a bit shaky. Playing with the core functionality might sound a bit scary.

There are abundance of options how to approach such scenario, but it looks a bit smelly again - even though we are "doing" versioning of the "API", seems like it's just versioning the surface, because our current architecture does not provide a way of easily versioning everything.

Do we have a feature toggle here? In a sense, yes. The decision is on the consumer side which "feature" should be used. Does this design supports easy "versionability"?

It depends on the range of change, but because we constrainted ourselves with risky work, affecting every single collaborator, seems like we are in a trouble.

What is missing?

This is just a toy example and I intentionally used a typical N-tier architecture, in which every layer contains related components: controllers are together, services too, etc.

But it seems that this example is missing similar properties as my introductory one. Distributed, hard to manage in terms of complete hideability/versionability.

There is no easy way to make entire functionality (by entire I mean all related components) truly versionable, except going to an every single guy and doing some work there to ensure versioning.

Traits like hideability, versionability, provided by feature toggles, might highlight weak design, inability to have safe, risky-limiting changes.

Feature toggle as a design tool

A need for toggling some functionality on and off might suggest keeping things together if they fulfill a given capability.

"Keeping related things together" might be also known as cohesion. Then, cohesive functionality seem to have hideability and versionability by default.

A conclusion might be that a design/code that is easy to be changed, should be easy to be hidden or versioned.

How can we check if our design adhears to such conclusion?

The fairy of immutable code

Let's have a small mental experiment.

Imagine that the evil fairy came and turned all of our code into an immutable code. You know what immutability is, right?

Existing immutable code cannot be modified. But actually there's a way to solve it - the only actions you can do is:

• copying a file/an entire directory
• deleting a file/an entire directory
• modyfing a new file
• renaming a file/an entire directory

In other words, one cannot go to the file, add a single "if" and check it in.

Instead of that, one would need to:

1. Copy entire file and name it properly, e.g. with *New suffix.
2. Add this single "if" in the new file.
3. Delete the "old" file.
4. Rename the new file by removing *New suffix.

Phew, that's exhaustive, isn't it?

But let's go back to the second point of such sequence, before deleting the "old" file. At some point in time, in our project, there are two versions of the same file (hence namespace/class/function/type, you choose) available.

Isn't it almost like a versioning? If we ensured unique naming of classes/functions/types/modules/namespaces/packages then we would be able to toggle between them (this would require another change on the higher level, somewhere else, of course!).

This teeny tiny example involved single "if", but what if there will be more classes/functions/types (meaning, files) required to be extended/altered to be accepted as a solution?

I can imagine traversing entire project, all those directories and then applying actions only available on immutable code.

A litmus test for high cohesion

Thinking in terms of immutable code, it seems to me that capabilities provided by set of highly cohesive parts should be easily versionable/replaceable/hideable just by copying entire directory and ensuring uniqueness of its subparts (files/classes/functions/namespaces/modules).

I believe that "vertical slices" are another name for highly cohesive subparts, supporting versionability/replaceability/removability.

If the code from Example time: say hi to Users.Backend! section became "immutable", there will be multiple rounds of "copy-modify-delete" steps.

With proper vertical structuring, we should be able to copy entire directory, do proper extensions/modifications and ensure we are able to either delete "old" or provide a way to toggle different versions.

A very cheap verification of having highly cohesive code would be to delete a single file, an entire directory (or entire project, in .NET world).

Number of errors and their nature should suggest and highlight weakpoints.

• Single class got affected?
• Multiple classes in a single directory, in the same project?
• Multiple classes across multiple directories, still in the same project?
• Multiple classes across multiple directories, across multiple projects?

Answer to the first question might indicate high cohesion of a given subsystem. Going further might underline losing the cohesion points (there's no such thing, I made this up, but I believe you got my point).

Sometimes "if" is just enough

Immutable code might sound a bit scary. The black, fairy magic looks dreadfully and would make our day to day job a nightmare.

With high probability, if you do not change almost everything related to a given capability you are working with, just an "if" will be enough to version your code or to hide it.

"Teaching" some higher level components/classes/modules how to decide which version to pick (or whether something should be hidden) should be simply doable. Pragmatism is the key here.

We could, of course, apply advanced strategies, e.g. by using strategy design pattern (yes, pun intended), but we won't get away from "if" statement/expression. At some point, you would need to place it somewhere.

Branching with "if" (or with pattern matching, if your language supports it), in the code, can open a door of hideability or versionability too.

What about branching in git?

I am glad that you asked (in fact I asked that, but let's pretend you did so).

With git branching you can have versioning, for free, not doing some magic stuff with immutable code or other tricks.

And that's true. However, git branches do not compose. You cannot have your lovely API running two branches of the same codebase on the single instance.

With branching done in the code (because this is in fact what versionability means - branching in the code), you can have it.

We cannot escape branching, but the question is where we do it. In the code you have massive amount of control. But there are various caveats.

Your code needs to be modular, cohesive and properly coupled. Without those properties it might be really challenging to get, e.g. proper versionability.

Having that in mind, it's easier to version things using git branches.

But what you are losing is feedback received from trying to obtain versionability/removability/hideability/replaceability traits.

Feedback about your design.

Feature branches

We are used to feature branches so that they can get reviewed and quality assurred.

But they delay feedback. Not only in terms of the integration, but in terms of the design too.

The assumption is they are short-lived, but I believe that we all experienced Pull-Requests, touching 50+ files, spreading across multiple directories, multiple projects (in .NET world).

They were approved with some low-impact improvements, getting "LGTM" eventually.

Let's come back to one of questions we posed in this article:

How easy it would be to hide/version entire feature (entire = everything related to it)?

In this particular context, hiding entire feature, especially when we think of our code in terms of immutable code, seems like a drudgery full of torment.

Imagine going to every single place of those 50+ files and doing copy, modify, rename, delete. Torture.

Wait, won't we duplicate the code?

Oh yes, we will!

I mean kind of. We will have TWO versions of the same functionality, available at the same time, but only some parts will be duplicated. But is it actually bad?

As with everything, there are trade-offs. You gain vast amount of control (e.g. you can apply fixes independently), but it requires discipline.

What discipline?

Again, keeping your code modular, cohesive and well-thought requires time, analysis, proper specification through tests, proper verification through tests (What I am talking about? See The "ambiguity" of TDD).

Such implicit branching via feature toggling demands a totally different mindset. Are you ready to use it?

From the first seconds of a new code, we would need to make sure everything is ready for the evil fairy of immutable code.

Below is visual comparison of different ways of doing branching.

With implicit branching, a higher level collaborator decides whether to use AThing.cs or AThingV2.cs

With feature toggling (implicit branching done in the code), you choose when it is an appropriate moment for deleting the "old" code.

No rush, let is sink. You have control.

You can leave two versions as long as it is needed.

You can easily revert back to the "old" version. Apply fixes to it.

Experiment. Play.

Suddenly, branching via feature toggling opened another door - being able to use Trunk Based Bevelopment.

There is no free lunch.

To gain this kind of control and shortening the feedback loop, we need to make our code ready for such approach.

All the software design heuristics and rules help achieving so, but we need to apply from the first seconds of our work.

In "The art of destroying software", Greg Young recommends writing small, modular code that makes itself easy to be deleted and replaced.

With such mindset, we can "hack and break things", because we are able to cancel our experiment.

"Just coding" won't give us such properties. All well-recognized traits like Encapsulation, Modularity, etc., increase the probability of achieving such state - making our code versionable/hideable/replaceable.

Changing through copying

Recently, I had a pleasure to perform a refactoring of some other component.

One of the bringing value goals was achieving better maintainability and higher quality.

Such vague and abstract words were supposed to be materialized through making it more testable, by pulling all dependencies to the most outer layer.

Simple as that.

In fact, we didn't start with feature toggling, but thanks to having isolated, pretty independent piece of code, we made it versionable by copying entire project.

Duplication? Yes.

Control? YES!

Fast Feedback? Yes.

Discipline? Yes!

Duplication isn't bad, as long as you can delete the duplicated parts at any time, without being stressed that things will be broken.

The "old" code gets deleted, the "new" code becomes the only version available.

Are you ready to delete your code?

Have fun - go and try deleting parts of your codebase: entire directories, single files, single classes/functions and observe what happens.

How many other parts of your system got affected?

This is really instant feedback. I don't think you will get quicker than that.

There are other approaches and tools opening a possibility to analyze your design, e.g. Your Code as a Crime Scene or Software Design X-Rays, but they require a bit more effort.

Still, they might be great next steps, after you can spot a problem with your code either by thinking how to provide full versionability or just by deleting some parts of the code.

Start with immutable code constraint

Next time, do an exercise.

Use the fairy of immutable code approach and think that you cannot modify existing code.

Introduce feature toggling as the first step and see how much effort is required to achieve complete versionability (complete = all code related to a given functionality).

Is it itchy?

Is it painful?

Congratulations, you found things to improve!

Same sh*t, different name

Did I invent something new? Nope, I am again standing on the shoulders of giants.

Modularity, encapsulation, right abstraction, high cohesion and low coupling (I like to call it "proper coupling") aren't new.

But I don't think there is a faster way to get feedback about your design. By deleting some parts of your code you don't need to write anything new, like tests (either verification or specification ones). It can be a simple and cheap litmus test to understand where are you in terms of modularity/cohesion spectrum.

It takes time to reflect on the information acquired, of course.

Be your own hero

Should you use "immutable code" approach for every development work you do? Yes, it is a dogma. You need to comply.

Should you always start with providing "feature toggle", even for a small feature? Yes my zealot.

Of course, I am joking. Do whatever is working for you and apply any tool or approach with the doubtful mindset.

I think a simple heuristic might be helpful:

• Are you introducing a small change, that isn't risky? Just go with the "feature branching" (git) approach by just changing the existing code.

• Do you want to integrate quickly and get feedback sooner? Try applying branching in the code with "if", either by hiding new (possibly broken and unfinished) functionality or choosing between "old" and "next" version.

• Are you working with core, critical path of your system? Use "immutable code" approach and copy all code related to entire feature, introduce change there.

  -> You can't get it so easily? You discovered something about your design. Use this insight. Try to move towards complete versionability, but assess how much effort is required.

• Doing some new, cool stuff? Start with providing feature toggle that will enable versioning entire functionality. This is a decent intro to high cohesion, proper coupling.

Please remember - take what is useful, discard what doesn't bring any value.

Starting with "feature toggle"-first mindset might call for the discipline, but it increases the probability of getting the reward, the holy grail of software design - high cohesion, proper coupling.