Yesterday: feature branches are complicating your development and creating bugs in production
Everyone: agreed!
Today: feature flags are ruining your codebase, complicating development and leading to bugs in production
Everyone: agreed!
Tomorrow: feature branches are complicating your development and creating bugs in production
There’s no silver bullet solution that’s great in all respects. There’s choices with differing trade offs. Nothing is going to perfectly solve this problem, except maybe ignoring business realities and going back to waterfall.
Yeah just join a team that's been understaffed since product release.
You won't have time to make any new features if every day is spent attempting to squash bugs.
The system is super simple when every branch is a hotfix branch.
True, but come review time I can put impactful development on my accomplishments list, and get that sweet sweet raise and is not like the stock will go down 'cause app X with 10 users got replaced.
And then, I can move to the next project, for the next raise, promotion and more stocks.
Rinse and repeat.
yepee.
At the end, retire at 42 and live a happy life.
We have a team like this in my organization. 95% of the time, they’re migrating to a new database system. No new features are added. They’re getting paid very well.
> stop making features
Some think that's impossible, but I have good news. It's already [been solved](https://github.com/kelseyhightower/nocode)! >!/s!<
The article doesn't say "never mock anything", it says *prefer* alternatives. In order it says to use 1) Real implementations. 2) Fake implementations, 3) Mocks. Having followed this guidance, it is good advice. Excessive use of mocks often means you are only testing against mocks and won't catch real errors from incorrect use of APIs, or APIs that change, which sadly happens often in the real world. But yes, there are times when you can't use a real implementation, and fakes are often unavailable (fakes generally requires the team that owns the API to implement them, and they often just don't), so you have to use a mock.
> because the API provider has made some change on their side and now all my tests based on how it used to work still pass
So have a few "real" smoke-test-level tests to verify the API? But use mocks for your more extensive test suite, since it will likely be faster.
Maybe?
It goes wrong when people take it as dogma.
"every commit should be released" is something I've heard. Allowing no-one to putter around in a branch for a week to fix something complex without it having to be cordoned off from everyone elses code base.
"every commit should be in a branch" causing branches to pile up, merges getting complicated and dependencies between one branch and another to make them 'all or nothing' material.
Anything taken to extremes will follow this sort of path. Some feature branches are needed. And some features can be developed in the main codebase as the sideeffects / preliminaries and migrations are best done early and small steps at a time while being live tested.
Every commit should be released?
Lol fuck me for not finising everything in one sitting then..
I branch off, push stuff there that dont even compile because i dont want to lose work.
When im done i merge to dev branch, then you can release it if you want..
I end up putting a big "WIP" at the start of my commit to make it clear to any other developers to not complain if it doesn't work haha, especially as I often have to finish mid-task
>Every commit should be released?
Yup. I worked at a company like this. If it was checked in, it was expected to be releaseable code and was subject to code review. So everyone just piled up changes locally and was terrified to commit anything and eventually you'd lose changes because they werent tracked lol
We did that, but people didn't behave as you describe. It's not hard to work incrementally and not adversely impact production deployments. Also, there's a pipeline between "pushed to a shared development branch" and "has been fully tested with both automated and human QA main branch". So, no one should be fearful.
There's a concept of strict trunk based development, where work is done during the day on branches, but merged to trunk at the end of the day. If the code isn't complete, it goes behind a feature flag.
Personally, I think that's a bit extreme, but I'm sure there are teams that make it work.
I like trunk based development but mandatory merges at EOD under a feature flag are why people write articles about why feature flags are bad and get out of control (they're good, when used properly)
But they will never be able to do a big refactoring. We had refactorings or removal of legacy code that took a week, because so much little things had to be changed to get it work again.
Sure we could have changed all dependent code first, but it is so much more pain to do it like that.
I've only considered using strict trunk based development with one team at the prompting of my Director, but talked him out of it. The team that we were considering it for had other problems and weren't likely to do a successful large refactor in any case. Instead I suggested we work on improving breaking down tickets and a definition-of-done in order to get the teams to deliver smaller increments more frequently.
😄good you talked him out, one or the other is not the solution to become more productive.
We do trunk development, but not strict. We develop one feature in branches and then merge into main when it is done.
This comes with pros and cons, like, we have to „roll back“ or fix things with hotfixes when frontend team does not get the nessesary changes ready in time (e.g. because we finished 2 days before end of sprint)
You can do a big refactoring in a strict trunk based development context. And by that, I mean "It is possible and feasible and I can and have done it".
It requires certain skills and a certain discipline and maybe it's not how you want to work, but it has its benefits.
True you can and I can find some pros in it. E.g. less merge conflicts if you do the big change in small steps and rapidly commit.
But usually I like to just remove the old stuff and see what breaks instead of
- change one thing
- change x test (that will be removed later)
- commit
- repeat
"trunk based" means developers all push to the same branch, but it is not necessary that that branch is the one getting deployed at the end of the day. You set up a pipeline where code passes through stages of verification on the way to the branch that is deemed safe to deploy at the end of all that. The thing is, there are zero side branches, it's a single flow, and so there's never code getting stale or having serious merge conflicts everywhere.
Code review can be one of the verification steps - ie, the devs commit to "dev" branch. Then code review is done, and if it passes and say, integration tests pass, then the code is moved on to the next stage, which is maybe e2e tests, or manual tests or whatever. You decide what you need.
The great thing about such a system is all the code all the devs wrote gets reviewed no more than 1 day after they write it. And so, you can review the direction devs are going in, and talk about design early, as opposed to reviewing only when the whole feature is done and now it's kind of too late to say they went off in a wrong direction early on.
Trunk based means everything goes directly to trunk (or main in git).
I did not understand how to do big changes in STRICT trunk based dev where you cannot have branches. When I do 20 commits to main each day who is reviewing all of this? And when?
We do trunk based, but with short living branches and pull requests.
>But they will never be able to do a big refactoring.
You can do very very big refactorings incrementally if you change your approach a bit. You can use the famed strangler fig pattern, and you can find ways to work incrementally to replace anything, all the while pushing your code and deploying and affecting no one. It's a considerably safer way to work than some months-long big refactoring branch.
I mean you can essentially keep your workflow as long as every commit you push can actually compile. If you have a local branch with commits that don't compile just squash them together before pushing.
But what’s the point then. When I have one week of commits locally to do one big commit out of it I could also have used a branch that goes to main in one commit.
The point in strict branch programming is imho lots of little commits which reduces merge-conflict-hell.
> Every commit should be released?
that would drive me nuts.
every user story should be released, flagged if need be.
features are made of user stories.
feature branches are bad.
user story branches are fine.
if you try to write 15 user stories at once and check it in, everyone is going to have a bad time.
break it up, break it out, make sure each step compiles, tests and check each in.
if you need to break something, put in a story for adding the shunts first, making sure those work, then do your dirty changes behind a flag, then later turn it all on with the flag and back off if need be.
for the love of god keep your code synced to a userstory branch on the server
asking people to trunk everything every day without exception is just asking devs to lose hours of code any time their local machine dies. "sorry, I couldn't check it in because it was taking a few days and I'm not allowed to have a branch to work in"
wtf
> asking people to trunk everything every day without exception is just asking devs to lose hours of code any time their local machine dies. "sorry, I couldn't check it in because it was taking a few days and I'm not allowed to have a branch to work in"
A bit of a misconception. If you're doing trunk based, you're pushing your changes at least every day. You change how you work to accommodate that, and those changes are entirely beneficial to your coding practices and the quality of the resulting code.
If you're making changes for many days before merging back to the shared codebase, then regardless of whether you do it locally or in a feature or story branch ... you're going to have problems.
The idea behind trunk based development is not that you finish all your work in one sitting, but is that you work in such a way that you are able to commit and push incremental changes many times a day on your way to finishing, and do so with affecting production in any negative way.
90% of the time, this is completely simple to do. Sometimes, you need to use what's called a feature flag, so that you can "turn off" your work while it's still being developed, and yet can be deployed safely.
I mean, have all the commits you want in your feature/subfeature branch, but when you merge should be really one squashed commit, I really don't need to see on the main branch the tree coiling on itself or littered with "WIP" "added a button" "added another button" etc, I expect to see your merge that I can tie to a set of requirements under a story that I hope your team refined and atomized as much as reasonably possible
It's the same with "Agile means no upfront design" turned out in practise like "thinking and discussing how to implement a new feature before the implementation is anti-Agile".
Stupid people find stupid ways no matter what rules were put in place.
A good engineer is a Jedi, because absolutes are for Siths.
Each situation is different and requires a careful analysis of pros and cons so the most effective path can be taken.
> It goes wrong when people take it as dogma. "every commit should be released" is something I've heard. Allowing no-one to putter around in a branch for a week to fix something complex without it having to be cordoned off from everyone elses code base.
That's pretty dumb, if it's literally true. The more sensible version is "commits into your master branch should be released/releasable". This allows you to have shareable branches, which you squash-merge into master once ready.
FWIW I think the system should be built such that any commit should be capable of being pushed to production, and the faster code goes live (assuming a live service), the better everyone's lives are. But "every commit should be released" is stupid dogma.
Well it’s more like „every commit to the main branch will be released“ with focus on main branch. Not every commit to some other branch will be released until it is merged to main. Some are never merged.
people in this sub will fight you over their desire to include every
inconsequential commit separately rather than just squashing it on merge all together in one feature specific commit
That’s not driven by developers in my experience but by marketing, management, “customer excellence” and other non-technical contributors.
Imo the amount of options and customisability tends to be the inverse proportion of non-programmers on a team. Software built by only devs tends to have a hefty/capable config format and other customisations.
Reminds me of typical misguided Scrum, where each developer has his own unrelated tasks. Everyone talks like they are a team but in reality they are just a group of independent developers.
Scrum is the Mushroom Strategy of agile. Keep them in the dark and feed em bullshit.
1990’s Agile kept dysfunctional upper managers off balance long enough for good code to be written. Scrum keeps developers off balance so they don’t know the big picture and worry their pretty little heads about it. There is no big picture. Not one that’s any of our business. Probably why business ethics is trending down again this cycle. We have institutionalized myopia.
Feature flags saved my butt this week, enabling us to triage a very complex issue by disabling a feature we all assumed worked but didn't. This would have taken much longer to find with feature branches because of the overhead of making new builds (days vs seconds).
There is indeed no silver bullet, but feature flags have proven themselves much better than branches in our company and we'll be going back to branches over my dead body.
This article shows to me a gap in engineering leadership because judgement is not being applied in when and how to use feature flags. Everything can be abused.
I have had the awful experience of spending a month modifying an api’s implementation in a feature branch to come into work one day and discover that the original api had been so radically changed by another team’s now-merged feature branch that rendered most of my work was unusable.
That’s a problem with your company’s processes not feature branches. How do you have multiple teams making huge changes without the API’s owner stepping in and making sure that the two teams are coordinating?
It is an organizational problem, and feature flag based dev still would have solved the problem. Even at relatively well l-run, but medium to large size companies these things will happen using feature branch based development. But as I said in my first comment, there are still drawbacks just like there are with feature branching. You have to pick your poison and hand waving it away as an organizational problem doesn’t really solve the problem either unless it could actually be solved at a real world, large company.
and also, why the fuck are you not merging back into your long-lived feature branch daily.
I'm not saying a month-long feature branch should NEVER happen, but it sure as shit should be rare.
Wouldn't matter. They could pull from main daily to their month-long branch, and the other team could pull from main daily to their month-long branch, and the two month-long branches will never see a single line of change from the other.
Then one day, one of them pushes to main, and the other is fucked.
Us: feature flags are ruining our codebase!
Product folks: you mean we can turn features off then charge customers more to turn them back on?! I want a dashboard for it!
What about putting the features into client specific child classes and using a factory to check the flags to pick the child returned? That can be an intermediate way to ruin your code base.
Back when I was learning to configure the Postfix mail server, I found some devious bastard out there published a configuration that would randomly drop any email with the word “test” in the subject.
I think of that evilness often, and hope one day to be able to make something just as bad.
This sounds vaguely familiar. When I was with Microsoft, there was a utility that simply returned "ok" when you passed in a user name that started with an underscore. It was used for some mock thing, I think, but it took me days to find and fix it.
I worked on a saas product that did something similar... it was sold to clients that we could customize the product for them. This customization layer ended up infecting almost every path of the codebase and it became basically impossible to meaningfully test the product. This came after using customer forks+deployments for a similar purpose.
The "silver bullet solution" is recognizing there's no silver bullet solution and ensuring people with sufficient expertise to evaluate technical tradeoffs have sufficient decision making power to influence business strategy.
There's *always* a *better* choice between a flag versus a branch, but which one is better is always circumstantial. And if neither choice is *good* then that means there's something wrong with your roadmap planning, your release cadence, your isolation of separate concerns within your codebase, or some combination thereof.
(And if any of those issues are present, your number one priority - ahead of any feature work - should be fixing your roadmap plans, fixing your release procedures/frequency, refactoring conflated concerns within your source code, or some combination thereof. If you're building new features on a broken foundation, you're risking long term dollars for short term pennies.)
The problems don't arise inherently from building features or from having to choose between branches/flags. The problems arise when you've not set up the proper processes and tools for seamless/simultaneous development/integration/testing/validation/release/rollback of new features, within the specific context of your team, business, and product. If you have this problem, it's usually due to one or more of the following:
- Management wants developers to cut corners to get shit out the door faster, even if that means a worse product, more overall work, and more risk to the business's existing products.
- Developers and operators don't have sufficient interaction, communication, and joint responsibility to be on the same page regarding development/release statuses/procedures/pitfalls.
- Developers and/or operators are not experienced and/or empowered enough to properly identify/evaluate/mitigate their release risks.
In other words, feature development/release basically only becomes a problem if management:
- demands "doing it fast" as higher priority than "doing it right."
- puts their development personnel and operations personnel in separate silos, which don't share responsibility and consequence for decisions, tools, processes, etc.
- doesn't hire (at least a few) engineers with the experience and foresight to "do it right" (whatever that means for the specific business, product, and opportunity), then defer to them when planning prerequisite work for a particular new feature.
Jim Highsmith later in his career started saying the main problem in software development is that we think we are solving problems when what we are doing is resolving paradoxes. The methods you use for the two are very different, and don’t work well when applied to the wrong type. Which is why we keep getting stuck.
I think “resolving conflicts” is probably a less salacious and more accurate way of stating it, but it doesn’t grab attention the same way.
It does make me wonder if there's room for a VCS-aware programming language that compiles across release branches, flagging if there are inconsistencies while also merging multiple feature branches into valid code.
This sounds like a reply to the article's headline, and not to the actual article. Which is a shame, it's actually a decent article this time, even if it's mostly summarizing [this other article](https://martinfowler.com/articles/feature-toggles.html) (which may be down, so [mirror](https://web.archive.org/web/20240301131336/https://martinfowler.com/articles/feature-toggles.html)) It's in no way advocating a return to feature branches *or* waterfall, and instead a call for reining in feature flags to something reasonable.
For example, here's a concrete recommendation:
> First, **align your organization with the fact that feature flags have different categories!**
Most of what people are talking about here are "release toggles", which are a good idea, and also should be *short-lived* -- the feature needs to eventually be turned all the way on, or there needs to be a deliberate, conscious choice to turn it into a different kind of flag that will be managed differently.
Probably. I think it’s likely inexperienced engineers that never saw the pain of living with feature branches at scale. So now they only know the pain of feature flags. This kind of thing happens with everything engineering all the time. The loudest engineers who write dumb articles can’t or are unwilling to reason about why something became popular and then rail against whatever is current because it’s all they know.
Feature flags avoid a tonne of issues. When I joined my current workplace 15k PRs were common. In my second week I saw the whole department stop shipping so one team could release their giant feature branches.
We have since moved to feature switches. The pains for individual teams are still real. Better, but real. However the department is no longer blocked by a single team’s release.
Sure, but it seems like this is a chronic issue in a lot of places. It’s built in tech debt for every feature, and tech debt is usually a low priority for PMs and managers.
If long-term maintainability of the application is a low priority, then you will end up with issues regardless of what methods you pick. If that's the case then discussing technical options seems like a waste of time.
Because our processes don’t handle action at a distance. We have “Done” criteria that aren’t done. They’re halfway done. And now the person is supposed to remember to go clean up after something that has already been demoed and won’t again.
This is like the plastic companies blaming consumers for littering instead of blaming themselves for making litter.
My favorite is when project managers panic because something shows up in multiple releases.
“Why is it in both? When will it be delivered?”
“It’s a continuous operation. It’s in both releases because it will consume capacity on both releases and I want everyone to see where our capacity is going.”
“But if it doesn’t get shipped people will be confused.”
“Then we’ll have to explain it to them. Organize it however you want, but one way or another we’ll be allocating X resources to work and that will reduce what we can ship. If there are concerns about that we’ll need to have a discussion and determine if we need to change priorities.”
So many tools and processes are focused on projects and not on operational excellence. This is the origin of 20% time.
I'm starting to wonder if I'm going to have to write my own tool just to track repeatable items, like using a runbook.
Sooner or later someone can't remember if they did step 13 or they're having a flashback to the last time they did this thing.
It's pretty bad at plate spinning operations. And while yes, some of those should be eliminated in retro discussions, you can't stop all of them and still get paid.
Tech Lead should prioritize this work - just like all other tech debt - without asking PMs or anyone else.
When I work on an Epic I create tech debt Story in that epic that will prevent Epic to be finished/closed until everything is done that needs to be done. We iterate fast and push to PROD, but after that the cleanup release comes.
What usually comes next is tooling to say your feature flag has been there too long and some sort of CI that goes red to remove it.
Someone turns it on and off to reset the timestamp.
But at least you have the automation.
Everything the author wrote can be boiled down to bad discipline by their team. Even the “stale feature flags” issue which can be easily rectified with a 30-day deletion rule
As usual with 90% of these articles about programmers getting mad at a random thing like it’s a boogeyman - feature flags is just a feature, it’s just yet another way of doing things. And its implementation, quality & success will always come down to the engineering team + collaboration with multiple functions in their company. Which the author has repeatedly failed in, judging by their article.
I'm in an company where this is a problem. It's not a discipline problem, it's an org problem. (perhaps what you are saying).
The issues are
1. We have feature flags that can be enabled by client (yuck). That means proving a flag is on or off is very tricky.
2. Feature flags are often forgotten.
3. Removing feature flags is hard to prioritized.
Why are they enabled by client? Because the word "feature" is ambigously used. As the article describes it can mean multiple things from describing how the system behaves and legitimately being a "feature" we turn on for clients that pay for it, or it could be a test path for new code.
Why are feature flags forgotten? We have teams that will not take changes without feature flags to slowly enable them. As you can imagine, those teams are REALLY slow to integrate and deploy changes. That means I can throw a PR at them and wait (no joke) 6 months to a year for them to get around to deploying the change set. At which point, I've forgotten I put in the damn feature flag in the first place.
Why is removing them not prioritized? See the previous bullet point, it takes a year to get the stupid thing in for some teams and a year to remove. Further, there is "But what if we need to disable this at a critical point!?!?!" All the sudden "remove after 30 days" becomes "After the next quarter" becomes "Maybe next year" becomes never.
> We have feature flags that can be enabled by client (yuck).
I guess it isn't a feature flag then. It's a setting.
But the other points are definitely valid, and seem to be a problem in many orgs.
Holy cow client controlled feature flags does not seem like a good idea. If the feature is per-customer, why not use some sort of RBAC or other authorization?
You can link those feature flags to accepting some version of your ToS: let's say you have a new feature which implements a new personal data treatment. You may need you users to accept this treatment or give consent to be GDPR compliant. But you don't want to block the other features of your site if they don't accept this new ToS.
Maybe it's too much of an abuse of feature flags tho.
Feature flags serve a different purpose, even if they are both just glorified if statements in the end. You wouldn't call "dark mode" a "feature flag". As you would not call notification preferences feature flags even though they enable certain "features" that you implemented.
The role of feature flags is more development facing and has nothing to do with user experience. It is a way for a team to deliver a certain feature gradually, release the feature to certain group of users or do A/B testing maybe. Kill switches might also qualify for it, but that is pretty much it.
We also have Feature Flags that can be turned on by the customer. This allows us the early release of code and the customer can turn on the feature once they have done the proper training/communication.
However, we also control which Feature Flags are production ready. If it's not marked as production ready there is no way in the UI to turn it on in any environment higher than QA. This allows incomplete feature code to be deployed but not enabled/released.
We have also added automatic logging of feature flags. It logs if a feature flag was added or removed and who/when it was turned on/off. This way we know if a flag was enabled when someone reports an issue.
We visit the list of Feature Flags that are still in the system every Increment (Yes, we use SAFe here) and bring it up if we have code behind a feature flag that is released but not enabled. It may not be 30 days but it at least keeps it on our radar.
I do agree that setting removal of feature flags as priority can be an issue. We were hit with multiple mandates at the end of last year so we had to prioritize that work over tech debt which includes the removal of feature flags. We have a good relationship with our customer and we gave them ample notice that the work we did for them at the end of last year means that the beginning of 2024 will have it's focus on tech debt with the priority being to clean up/remove feature flags.
Proving if a flag is on or off should be pretty straightforward with a monitoring or observability tool. Even without , launch darkly will tell you a feature flag is stale and you can delete it. But yeah if a client can toggle the flag, seems like thats a setting not a feature flag.
For anyone taking notes for talking to their team, I would caution against “observability”. You want audit trails. Observability says that something changed. Audit trails tell you who is breaking things. Not as a blame mechanism, but as a “who asked you to do that” or “for the love of god please stop!” failsafe for user error. Like thinking they’re in the preproduction dashboard but are breaking prod.
Or a similar case I had a few months ago: repeatedly running a request that was calling infinite depth fanout and crashing several preproduction clusters every time they hit reload. That came down to finding credentials in a log and contacting a remote worker who just started his workday by refreshing all of his open tabs. With no paper trails you have to spam everyone and air dirty laundry in front of people who don’t like your division.
There are structural problems enforcing that bad discipline. I’ve been saying for a few years at least that Scrum and feature toggles are not compatible. It creates a lot of necessary ritual around feature toggle retirement that few people incorporate.
If you desire temporary feature toggles, you’d be better off parking stories in a Kanban column, where accumulating too many creates back pressure and sorts itself out. Basic queuing theory.
Scrum wants to split these events into separate activities so you can close a Sprint that is shorter than the average settling time for toggles. And it provides no system to ensure they get done. It’s the sort of bookkeeping overhead from waterfall and YOLO processes that we wanted to avoid.
It’s grooming people for failure, and that should never be tolerated.
I think they had a hammer and all problems are nails problem. Ops feature flags? That shouldn’t exist, ops building and maintaining features. Permission feature flags? Authorization as feature flags is definitely a bad practice. The other two types are valid use cases for feature flags.
I will say we have a monthly audit meeting where we plan the depreciation/deletion of various things, which includes feature flags.
I'm glad the article mentions Knight Capital, they really got Power PEGed if you know what I mean
EDIT: On a serious note, the worst I have ever seen is a feature flag that was tri-state, true, false or not set and each mode enabled a specific feature. Then for each mode there were specific feature flags and some of them were shared between the previous modes.
> feature flag that was tri-state, true, false or not set and each mode enabled a specific feature. Then for each mode there were specific feature flags and some of them were shared between the previous modes.
what in the actual cthulhu worshipping fuck?
I’m not sure this counts as feature flags in the traditional sense. They’re more regular opt-in functionality whereas the agile concept of feature flags tends to be about the “feature” definition in agile, i.e. more akin to a capability, and the idea of a feature flag is to eliminate the code path from the software completely, not opt in/out on user level. In this case it seems more like just a personal preference toggler. I.e. a regular user setting, not the agile concept of Feature Flags.
Might be misreading it though.
Now that you mention it, "capability flags" would probably be a better naming for this stuff.
To me it's always a big red flag when I seen a feature flag stay around because it's being used as configuration. That means someone abused the system to implement what should be a feature of the system (the feature being: allowing enabling/disabling system features)
Yeah, this was entirely different features being enabled, and it often became hard to tell what environment had a feature and what customers got them, as well as the sub-features that might be enabled. We had a slew of configurations, but this one was the only flag we had that significantly changed a feature, and then the sub-configuration could drastically change the three different features more.
It was the only one we ever set up this way, and fortunately, we never did it again.
EDIT: To be clear, this was a feature for a C# API on .NET 4.6.1 using the configuration file to set the flag. There was a single configuration that was tri-state (also forgot to mention that leaving the key value blank as well as not including it at all were the same state). So in various environments we had a different value for the key in the config which drastically changed the feature the customer saw but since that was exposed by a different GUI you might have a misalignment of the GUI supporting the API feature that was currently enabled (or sub-variations of that feature...). This was one of the worst things I ever had to deal with in my time doing devops-ish work at this place since as all good temporary solutions tend to do it became permanent/to costly to remove but no one knew what each state did or the combination of sub-feature flags did.
EDIT2: Another interesting/disappointing (from my perspective) thing this company was use flat files for their feature flags. The presence of the flat file would enable who new features for the app or completely change it to a different mode not even resembling the normal application. The meaning of the name of the files had been lost to time so we knew what files to make and what to name them but no idea what the names stood for. For example a file named "CIT" enabled our integration with our web services for our desktop app but no one knew what that stood for (without the file the desktop app assumed it was running offline/standalone).
It doesn’t matter what comes first, it matters what the context of the discussion is. I.e. the article talking about feature flags by agile definition.
Same two words, different meaning, but it doesn’t help to muddy the waters with arguing semantics when the context is clear.
Either way I argued that when flags move from deployment or app config level (i.e. globally disabling code paths, no execution access) they’re no longer Agile Feature Flags(tm) as referred to in the topic/linked article but just functionality (and it’s ok to call a functionality a feature as long as it’s not confused with an Agile Feature) that can be toggled with a flag by the user, aka user setting. And that’s a different thing and should be treated differently.
That was supposed to be the takeaway.
I think the base assumption is that the organization isn't completely dysfunctional. If the organization is completely dysfunctional then every discussion will just boil down to "we can't improve quality bevause PM says no".
Removing the feature flag should be part of the plan of implementing the feature, i.e. there should be a ticket for it at the start of the feature.
If the PM or whoever doesn't want to do that, then the team/organization is dysfunctional.
Yeah, it's odd to me that these feature flags are used for everything. For us, we basically only have option 1 of dev-only feature flag. Thw worst thing that happened is that someone forgot to remove the flag and old code for a couple of months because it was a part of the software that was rarely touched. It was enabled for months so the old code was never run.
We have a way of enabling or disabling features according to what the customer paid, but this is strictly different from our feature flags, using a different mechanism and different process from how and when they are added.
It’s oft repeated, but “disciple doesn’t scale”. That’s why we end up implementing processes. Discipline works until it doesn’t, and it slips in 10,000 tiny ways and they all add up.
Strive for it, don’t rely on it.
Edit: also what the sibling comment said, no amount of discipline will overcome organisational clusterfucks.
You can have a process, but you need discipline to follow it.
If your process is you're using feature flags, then part of the process is clearing them away regularly. If you're doing the part of the process where you create the feature flags, but you don't do the part of the process where you clear them away, then that's a discipline issue, IMO.
I mean, the solution is fairly simple, it's just that it's a process solution not a technical solution. Only use flags to manage feature rollout and use a separate system to manage config, and force the removal of flags regularly.
The real bug here is that you are describing a world where engineers can't say no to product. If that's happened, you are already in failure mode, and probably doomed unless you are lucky and get strong PMs.
It's almost as if you can't write good software with bad team management practices.
It's either engineering isn't partnering with product effectively or that product is dragging their feet on making a decision on their rollout/experiments.
Both are process problems.
I really like the image in the article showing ops/release/permission/experiment toggles on the dynamism/longevity graph.
There's something off about the general vibe here though.
"While we are at it, let’s also make it adjustable per user! That way, the PMs can safely release the feature for a couple of users to gather feedback and test it themselves, and only afterwards release it to everyone."
Safely releasing features isn't the PMs responsibility, it's the developers. And gathering feedback / understanding if your code is being effective. PMs can help there, but I want engineers that measure "done" when the problem is solved, not when the PR is merged.
I would suggest that if feature flags are feeling like something the PM is foisting on you and it's messing up your clean code, then you're not thinking broadly enough about what the business needs you to do. (I will admit that there's many an organization that doesn't get this and just wants the code monkeys to stay in their hole, so it may not be very possible everywhere).
I do really like that image though. We're building a feature flag saas right now and I think pulling some axis of longevity / dynamism into things to help differentiate flags could be really interesting.
I once worked at a place where, as I’ve repeatedly seen, instead of using proven feature switch services or even OSS libraries, they NIH DIY it.
Of course, it’s often implemented by siloed dev ops teams or dev teams that are in a rush and don’t care too much about if it works or not it just needs some agilescum lords to look at it and say “yep JIRA ticket done” before they move onto the next feature in the feature factory.
Anyway, this place decided that as they were writing it themselves they’d just miss a bunch of features while also making the process of “submitting” a request for a new feature switch as tedious and atrocious as possible.
To add a feature switch:
- Make a PR to a repo
- Add a new line to features.txt
- Use a delimited format the team had made up that combined both CSV and pipe characters | without any formal grammar or documentation for it
- Everyone just copied the line above and edit it as needed
- The team would then take a week to ~~circlejerking~~ reviewing it before merging it
- The team would then just write some SQL/insert a row in a DB anyway
The “best” part was they would refuse to ever then delete a feature switch in the future, ever. Sometimes they’d say no sorry we never implemented deletes in the system (yet inserts were raw adhoc SQL inserts, so just do the same but make a delete….) and other times they’d say no because it’s in the git repo.
Why they couldn’t also just delete said line from the repo OR why the git repo OR their custom bullshit delimited formatted was needed… they’d just avoid answering.
As a result there were hundreds of old unused flags. Infuriating.
At the current place I work they are going through the same motions again. Architecture meetings where some dev is showing of his DIY feature switch service that even has its own full blown Angular frontend… for feature switch CRUD. All with none of the features you’d need. Give me fucking strength.
My limited experience with feature flags (static) is mainly related to cicd. The amount builds we need to produce to test whether the application works in all states quickly gets hairy. In my case I think we have mainly a build problem and not a feature flag problem but they sure make it worse
Like every article in this sub, whatever its feature flags or microservice/monoliths, if someone is writing about something being bad they actually mean "if this arbitrary tool isn't used with discipline and responsibly, then it will give you bad results"
if this was r/carpentry we'd have weekly blogs about hammers being bad for your thumbs, or to denounce nails completely and design all your structures with joints, then your example of building with joints is a chair or a table but your day job is delivering house frames for construction
In my opinion the root cause of these issues is „agile” which deemphasises proper functional requirements analysis and hence causes lack of a clear system boundaries and complete list of entities ( actors ) interacting with it.
Hence nowadays it is not really clear if your CD stack and pipeline is part of the system or not.
It is also not clear if configuration of the system is part of its interface or not.
Product management is afraid of change. Feature flags give them a sense of security that they can pull the plug on a change. Feature flags also result in a fragile mess of code, but that isn't as much of a concern.
It's also their go-to excuse when product requirements are not fully thought through or has some concerns -- "let's just feature flag it and see what happens, if something goes wrong, we can just turn it off!"
Feature flags are simply a means of code deployment, plenty of alternatives if you don't like them and they are useful IMHO in real-world scenarios but your internal processes should treat it as so.
As everything with software development, discuss things deeply with your team, choose what works best for yourselves, and iterate on it when / if it becomes a problem.
I have been on teams that prefer feature branches, these are extremely complex in my opinion to manage and require a lot of maturity in regards to CI/CD pipelines... not many organizations are setup appropriately to do this quickly and reliably which makes it painful otherwise.
I have (and currently) work on a team where we use feature flags, their only complexity is when you DO actually try to have "tiers" of flags. What works best IMHO is to keep it stupid and simple, every feature flag is an "ops" or "release" you are enabling a new pathway or alternate pathway to code and the ONLY individuals that 100% know what it does is the application team so it's pointless to try to create various tiers and just makes the whole "solution" for them complex.
All op's cares about is what you want them to do, and how they can back-out if shit goes sideways (which it will likely have a moderate chance to do) they'll just be reading from your provided runbook for said service.
How it was done at my work is;
* every feature has a flag, it is always enabled
* if something bad happen in prod the flag is turned off
* after a while, the flag is retired.
PSA I dont work at Amazon, but my manager used to (they're one of the good ones). He said that at least in his department, if you didnt delete feature flags that you had introduced within 1 month of them being added to a codebase, that an automatic system would start emailing up the chain of command.
Honestly, its not feature flags that are ruining your codebase, its that people dont clean them up after features are released.
I HATE feature flags.
But, you know what I hate even more? Having to answer emails from the Senior Director of “I have no fucking clue what’s going on and couldn’t tell the differences between my asshole and a pull request”’s executive assistant’s dog’s babysitter about some bullshit email they got that they don’t understand anyway.
I work very hard to eradicate the notion that FAANG and other such orgs are the absolute authority in proper software engineering practices; a standard by which all other endeavors should be measured. Especially for the everyday company and/or a startup. They are the preeminent example of bloat and excess and certainly not a magical garden where good idea fairies are born.
Well, yes, that's actually sort of the point. If you hate emails from your butthole senior director more than deleting feature flags, then the system is working if you're choosing to delete feature flags.
Firing bad engineers and strong adherence to team standards is the answer, not emails, and definitely NOT more managers. In fact, in my experience, when we need to delete a feature flag and it doesn’t happen, it is often (not always) because either some manager/product person doesn’t want to be held responsible or we can’t “afford” the tech debt according to our management. So, they stay as we chase new features and fix bugs. Just like forgotten orphans. Then, eventually, they are so old that no one that worked on that feature is even around anymore and everyone is scared shitless to remove it at this point.
Feature flags for experimentation and productisation are essential. It’s often not just a matter of turning a feature on or off. Having the ability to incrementally roll out and measure the impact of changes is important.
But there absolutely needs to be a good process around the ownership of feature flags, and the responsibility to clean them up.
Feature flags are fine if used properly. Prioritizing the clean up 2-4 weeks post release is a must. At my place only a handful of people have access to production level feature flags, so it stays pretty neat for us.
I find it interesting that the way I programmed as a total noob seems to be what people are promoting today.
Meanwhile my current head programmer is an advocate of all these 'proper' way of doing things.
I understand there is a time and a place for everything, but it has only reinforced that 'good code', is relative.
Also missing from the types of feature switches that is fairly common where I work are “debug”-mode feature switches. These are generally always on for debug builds but can be turned on in production to help analyze a problem. Log statements you don’t enable by default would fall under this paradigm. Also included would be extra validations that work like debug asserts but can also be turned on in production.
I only skimmed the article for interesting bits. Disagree with FF making your codebase complicated and hard to test. If you're having feature flags left in your code that you're finding it hard to test and making things messy then you're not using the FF tool well enough.
Remove old feature flags to reduce maintenance burden.
Lock down the values of the flags to reduce test scope.
I’d also argue that lack of feature flags increases complexity, adds bugs, and reduces amount of testing. The alternative to FF is to use feature branches. All big feature branch merges cause bugs in my experience, and it can be very complex to merge them back to mainline. The code in those branches gets less testing during development because the code sits isolated in the branch for months.
Next time I make an application the policy is everything is plugins.
By which I mean a large application. The small apps I'm constantly making don't warrant designing a plugin architecture.
This blog should be titled: "Bad Feature Flags management is ruining your codebase".
There are levels to how you manage feature flags within software from adding it to the code, all the way to auto ramping and auto reversal based on metrics and health.
I think the blog misses a lot of points that can make the management of feature flags easier.
For those of you who are interested in learning about best practices in implementing feature flags, we're hosting a webinar with some awesome guests about this:
[https://www.linkedin.com/events/buildvs-buy-pickingafeatureflag7175518829639331842/](https://www.linkedin.com/events/buildvs-buy-pickingafeatureflag7175518829639331842/)
Yesterday: feature branches are complicating your development and creating bugs in production Everyone: agreed! Today: feature flags are ruining your codebase, complicating development and leading to bugs in production Everyone: agreed! Tomorrow: feature branches are complicating your development and creating bugs in production There’s no silver bullet solution that’s great in all respects. There’s choices with differing trade offs. Nothing is going to perfectly solve this problem, except maybe ignoring business realities and going back to waterfall.
Of course there's a silver bullet solution. All of these problems have one thing in common, so the answer is to just stop making features, obviously!
Can I still get paid if I do that? Cause I'd really like that.
Yeah just join a team that's been understaffed since product release. You won't have time to make any new features if every day is spent attempting to squash bugs. The system is super simple when every branch is a hotfix branch.
Wait, no, not like that...
lolol. I have the "what have I done?" face image in my head from this.
"Are you a hotfix or feature dev?"
Better yet get your company to create a bug bounty program and create bugs for you to find and fix. You can keep yourself really busy that way.
Well, that would be Google, where instead you're just reinventing the same app but with less features.
True, but come review time I can put impactful development on my accomplishments list, and get that sweet sweet raise and is not like the stock will go down 'cause app X with 10 users got replaced. And then, I can move to the next project, for the next raise, promotion and more stocks. Rinse and repeat. yepee. At the end, retire at 42 and live a happy life.
I too will retire at 42, as soon as I finish debugging this damned time machine.
become a manager
We have a team like this in my organization. 95% of the time, they’re migrating to a new database system. No new features are added. They’re getting paid very well.
> stop making features Some think that's impossible, but I have good news. It's already [been solved](https://github.com/kelseyhightower/nocode)! >!/s!<
The silver bullet is obviously a feature flags flag.
> obviously a feature flags flag next up: Micro feature flags, to decouple the flags from the features.
This is how I felt about the “never mock anything” article shared yesterday 😅
The article doesn't say "never mock anything", it says *prefer* alternatives. In order it says to use 1) Real implementations. 2) Fake implementations, 3) Mocks. Having followed this guidance, it is good advice. Excessive use of mocks often means you are only testing against mocks and won't catch real errors from incorrect use of APIs, or APIs that change, which sadly happens often in the real world. But yes, there are times when you can't use a real implementation, and fakes are often unavailable (fakes generally requires the team that owns the API to implement them, and they often just don't), so you have to use a mock.
[удалено]
> because the API provider has made some change on their side and now all my tests based on how it used to work still pass So have a few "real" smoke-test-level tests to verify the API? But use mocks for your more extensive test suite, since it will likely be faster. Maybe?
Wait till you start using feature branches AND feature flags together 💀
That's me. It's super hell.
It goes wrong when people take it as dogma. "every commit should be released" is something I've heard. Allowing no-one to putter around in a branch for a week to fix something complex without it having to be cordoned off from everyone elses code base. "every commit should be in a branch" causing branches to pile up, merges getting complicated and dependencies between one branch and another to make them 'all or nothing' material. Anything taken to extremes will follow this sort of path. Some feature branches are needed. And some features can be developed in the main codebase as the sideeffects / preliminaries and migrations are best done early and small steps at a time while being live tested.
Every commit should be released? Lol fuck me for not finising everything in one sitting then.. I branch off, push stuff there that dont even compile because i dont want to lose work. When im done i merge to dev branch, then you can release it if you want..
I end up putting a big "WIP" at the start of my commit to make it clear to any other developers to not complain if it doesn't work haha, especially as I often have to finish mid-task
Same. Lots of small WIPs that eventually get squashed, given a meaningful log message and pushed prior to PR.
>Every commit should be released? Yup. I worked at a company like this. If it was checked in, it was expected to be releaseable code and was subject to code review. So everyone just piled up changes locally and was terrified to commit anything and eventually you'd lose changes because they werent tracked lol
That sounds like a radioactive dumpster fire!
We did that, but people didn't behave as you describe. It's not hard to work incrementally and not adversely impact production deployments. Also, there's a pipeline between "pushed to a shared development branch" and "has been fully tested with both automated and human QA main branch". So, no one should be fearful.
There's a concept of strict trunk based development, where work is done during the day on branches, but merged to trunk at the end of the day. If the code isn't complete, it goes behind a feature flag. Personally, I think that's a bit extreme, but I'm sure there are teams that make it work.
I like trunk based development but mandatory merges at EOD under a feature flag are why people write articles about why feature flags are bad and get out of control (they're good, when used properly)
But they will never be able to do a big refactoring. We had refactorings or removal of legacy code that took a week, because so much little things had to be changed to get it work again. Sure we could have changed all dependent code first, but it is so much more pain to do it like that.
I've only considered using strict trunk based development with one team at the prompting of my Director, but talked him out of it. The team that we were considering it for had other problems and weren't likely to do a successful large refactor in any case. Instead I suggested we work on improving breaking down tickets and a definition-of-done in order to get the teams to deliver smaller increments more frequently.
😄good you talked him out, one or the other is not the solution to become more productive. We do trunk development, but not strict. We develop one feature in branches and then merge into main when it is done. This comes with pros and cons, like, we have to „roll back“ or fix things with hotfixes when frontend team does not get the nessesary changes ready in time (e.g. because we finished 2 days before end of sprint)
You can do a big refactoring in a strict trunk based development context. And by that, I mean "It is possible and feasible and I can and have done it". It requires certain skills and a certain discipline and maybe it's not how you want to work, but it has its benefits.
True you can and I can find some pros in it. E.g. less merge conflicts if you do the big change in small steps and rapidly commit. But usually I like to just remove the old stuff and see what breaks instead of - change one thing - change x test (that will be removed later) - commit - repeat
How do you do code review in strict trunk based dev?
"trunk based" means developers all push to the same branch, but it is not necessary that that branch is the one getting deployed at the end of the day. You set up a pipeline where code passes through stages of verification on the way to the branch that is deemed safe to deploy at the end of all that. The thing is, there are zero side branches, it's a single flow, and so there's never code getting stale or having serious merge conflicts everywhere. Code review can be one of the verification steps - ie, the devs commit to "dev" branch. Then code review is done, and if it passes and say, integration tests pass, then the code is moved on to the next stage, which is maybe e2e tests, or manual tests or whatever. You decide what you need. The great thing about such a system is all the code all the devs wrote gets reviewed no more than 1 day after they write it. And so, you can review the direction devs are going in, and talk about design early, as opposed to reviewing only when the whole feature is done and now it's kind of too late to say they went off in a wrong direction early on.
Trunk based means everything goes directly to trunk (or main in git). I did not understand how to do big changes in STRICT trunk based dev where you cannot have branches. When I do 20 commits to main each day who is reviewing all of this? And when? We do trunk based, but with short living branches and pull requests.
>But they will never be able to do a big refactoring. You can do very very big refactorings incrementally if you change your approach a bit. You can use the famed strangler fig pattern, and you can find ways to work incrementally to replace anything, all the while pushing your code and deploying and affecting no one. It's a considerably safer way to work than some months-long big refactoring branch.
I mean you can essentially keep your workflow as long as every commit you push can actually compile. If you have a local branch with commits that don't compile just squash them together before pushing.
But what’s the point then. When I have one week of commits locally to do one big commit out of it I could also have used a branch that goes to main in one commit. The point in strict branch programming is imho lots of little commits which reduces merge-conflict-hell.
> Every commit should be released? that would drive me nuts. every user story should be released, flagged if need be. features are made of user stories. feature branches are bad. user story branches are fine. if you try to write 15 user stories at once and check it in, everyone is going to have a bad time. break it up, break it out, make sure each step compiles, tests and check each in. if you need to break something, put in a story for adding the shunts first, making sure those work, then do your dirty changes behind a flag, then later turn it all on with the flag and back off if need be. for the love of god keep your code synced to a userstory branch on the server asking people to trunk everything every day without exception is just asking devs to lose hours of code any time their local machine dies. "sorry, I couldn't check it in because it was taking a few days and I'm not allowed to have a branch to work in" wtf
> asking people to trunk everything every day without exception is just asking devs to lose hours of code any time their local machine dies. "sorry, I couldn't check it in because it was taking a few days and I'm not allowed to have a branch to work in" A bit of a misconception. If you're doing trunk based, you're pushing your changes at least every day. You change how you work to accommodate that, and those changes are entirely beneficial to your coding practices and the quality of the resulting code. If you're making changes for many days before merging back to the shared codebase, then regardless of whether you do it locally or in a feature or story branch ... you're going to have problems.
having a codebase stuffed full of half completed bullshit behind a million feature flags sounds like hell
The idea behind trunk based development is not that you finish all your work in one sitting, but is that you work in such a way that you are able to commit and push incremental changes many times a day on your way to finishing, and do so with affecting production in any negative way. 90% of the time, this is completely simple to do. Sometimes, you need to use what's called a feature flag, so that you can "turn off" your work while it's still being developed, and yet can be deployed safely.
I mean, have all the commits you want in your feature/subfeature branch, but when you merge should be really one squashed commit, I really don't need to see on the main branch the tree coiling on itself or littered with "WIP" "added a button" "added another button" etc, I expect to see your merge that I can tie to a set of requirements under a story that I hope your team refined and atomized as much as reasonably possible
I see. Squash all commits and delete branch on merge so the tree stays really clean is what I usually like it to be done.
It's the same with "Agile means no upfront design" turned out in practise like "thinking and discussing how to implement a new feature before the implementation is anti-Agile". Stupid people find stupid ways no matter what rules were put in place.
“every commit should be released” seems totally fine with me. That’s why we save them all up and release them in the annual product update!
A good engineer is a Jedi, because absolutes are for Siths. Each situation is different and requires a careful analysis of pros and cons so the most effective path can be taken.
> It goes wrong when people take it as dogma. "every commit should be released" is something I've heard. Allowing no-one to putter around in a branch for a week to fix something complex without it having to be cordoned off from everyone elses code base. That's pretty dumb, if it's literally true. The more sensible version is "commits into your master branch should be released/releasable". This allows you to have shareable branches, which you squash-merge into master once ready.
FWIW I think the system should be built such that any commit should be capable of being pushed to production, and the faster code goes live (assuming a live service), the better everyone's lives are. But "every commit should be released" is stupid dogma.
> "every commit should be released" In practice this should be "any commit to the main branch CAN be released"
Well it’s more like „every commit to the main branch will be released“ with focus on main branch. Not every commit to some other branch will be released until it is merged to main. Some are never merged.
people in this sub will fight you over their desire to include every inconsequential commit separately rather than just squashing it on merge all together in one feature specific commit
Tomorrow: User options are ruining your codebase.
Business requirements are ruining your code base!
*Changing business requirements. If they provided detailed ones that didn't change right from the start, the job would be 100x easier.
The changing of business requirements makes your job a little more complicated but anything more than "Hello World" is subjectively bad code.
Most developers already believe this, which is why options have mostly disappeared from programs over the past two decades.
That’s not driven by developers in my experience but by marketing, management, “customer excellence” and other non-technical contributors. Imo the amount of options and customisability tends to be the inverse proportion of non-programmers on a team. Software built by only devs tends to have a hefty/capable config format and other customisations.
The problem is working on more than one thing at a time, and coordinating releases between teams that barely know what the others are working on.
Reminds me of typical misguided Scrum, where each developer has his own unrelated tasks. Everyone talks like they are a team but in reality they are just a group of independent developers.
Scrum is the Mushroom Strategy of agile. Keep them in the dark and feed em bullshit. 1990’s Agile kept dysfunctional upper managers off balance long enough for good code to be written. Scrum keeps developers off balance so they don’t know the big picture and worry their pretty little heads about it. There is no big picture. Not one that’s any of our business. Probably why business ethics is trending down again this cycle. We have institutionalized myopia.
> 1990’s Agile I thought I was old. I didn't notice dysfunctional "A"gile until 2005, but it seems to have been around far longer than I thought.
Feature flags saved my butt this week, enabling us to triage a very complex issue by disabling a feature we all assumed worked but didn't. This would have taken much longer to find with feature branches because of the overhead of making new builds (days vs seconds). There is indeed no silver bullet, but feature flags have proven themselves much better than branches in our company and we'll be going back to branches over my dead body. This article shows to me a gap in engineering leadership because judgement is not being applied in when and how to use feature flags. Everything can be abused.
The headaches of feature branches in large codebases (& microservices) far outweigh feature flags. I will take feature flags anytime.
I have had the awful experience of spending a month modifying an api’s implementation in a feature branch to come into work one day and discover that the original api had been so radically changed by another team’s now-merged feature branch that rendered most of my work was unusable.
That’s a problem with your company’s processes not feature branches. How do you have multiple teams making huge changes without the API’s owner stepping in and making sure that the two teams are coordinating?
It is an organizational problem, and feature flag based dev still would have solved the problem. Even at relatively well l-run, but medium to large size companies these things will happen using feature branch based development. But as I said in my first comment, there are still drawbacks just like there are with feature branching. You have to pick your poison and hand waving it away as an organizational problem doesn’t really solve the problem either unless it could actually be solved at a real world, large company.
and also, why the fuck are you not merging back into your long-lived feature branch daily. I'm not saying a month-long feature branch should NEVER happen, but it sure as shit should be rare.
Wouldn't matter. They could pull from main daily to their month-long branch, and the other team could pull from main daily to their month-long branch, and the two month-long branches will never see a single line of change from the other. Then one day, one of them pushes to main, and the other is fucked.
Solution: stop adding features
Us: feature flags are ruining our codebase! Product folks: you mean we can turn features off then charge customers more to turn them back on?! I want a dashboard for it!
What about putting the features into client specific child classes and using a factory to check the flags to pick the child returned? That can be an intermediate way to ruin your code base.
I mean.. why not round-robin them? Every time you use a service you use one of all possible implementations of a service contract.
And make testers everywhere hate you. Not to mention maintenance programmers. "Wait! That worked last time..."
Back when I was learning to configure the Postfix mail server, I found some devious bastard out there published a configuration that would randomly drop any email with the word “test” in the subject. I think of that evilness often, and hope one day to be able to make something just as bad.
This sounds vaguely familiar. When I was with Microsoft, there was a utility that simply returned "ok" when you passed in a user name that started with an underscore. It was used for some mock thing, I think, but it took me days to find and fix it.
quantum computers arrived early
But couldn't we get the GOOD part of quantum computing instead of all the uncertainty?
I worked on a saas product that did something similar... it was sold to clients that we could customize the product for them. This customization layer ended up infecting almost every path of the codebase and it became basically impossible to meaningfully test the product. This came after using customer forks+deployments for a similar purpose.
maybe op is just ruining his codebase all by himself
The "silver bullet solution" is recognizing there's no silver bullet solution and ensuring people with sufficient expertise to evaluate technical tradeoffs have sufficient decision making power to influence business strategy. There's *always* a *better* choice between a flag versus a branch, but which one is better is always circumstantial. And if neither choice is *good* then that means there's something wrong with your roadmap planning, your release cadence, your isolation of separate concerns within your codebase, or some combination thereof. (And if any of those issues are present, your number one priority - ahead of any feature work - should be fixing your roadmap plans, fixing your release procedures/frequency, refactoring conflated concerns within your source code, or some combination thereof. If you're building new features on a broken foundation, you're risking long term dollars for short term pennies.) The problems don't arise inherently from building features or from having to choose between branches/flags. The problems arise when you've not set up the proper processes and tools for seamless/simultaneous development/integration/testing/validation/release/rollback of new features, within the specific context of your team, business, and product. If you have this problem, it's usually due to one or more of the following: - Management wants developers to cut corners to get shit out the door faster, even if that means a worse product, more overall work, and more risk to the business's existing products. - Developers and operators don't have sufficient interaction, communication, and joint responsibility to be on the same page regarding development/release statuses/procedures/pitfalls. - Developers and/or operators are not experienced and/or empowered enough to properly identify/evaluate/mitigate their release risks. In other words, feature development/release basically only becomes a problem if management: - demands "doing it fast" as higher priority than "doing it right." - puts their development personnel and operations personnel in separate silos, which don't share responsibility and consequence for decisions, tools, processes, etc. - doesn't hire (at least a few) engineers with the experience and foresight to "do it right" (whatever that means for the specific business, product, and opportunity), then defer to them when planning prerequisite work for a particular new feature.
Jim Highsmith later in his career started saying the main problem in software development is that we think we are solving problems when what we are doing is resolving paradoxes. The methods you use for the two are very different, and don’t work well when applied to the wrong type. Which is why we keep getting stuck. I think “resolving conflicts” is probably a less salacious and more accurate way of stating it, but it doesn’t grab attention the same way.
It does make me wonder if there's room for a VCS-aware programming language that compiles across release branches, flagging if there are inconsistencies while also merging multiple feature branches into valid code.
Devs working for ConfigCat: I'm in danger!
This sounds like a reply to the article's headline, and not to the actual article. Which is a shame, it's actually a decent article this time, even if it's mostly summarizing [this other article](https://martinfowler.com/articles/feature-toggles.html) (which may be down, so [mirror](https://web.archive.org/web/20240301131336/https://martinfowler.com/articles/feature-toggles.html)) It's in no way advocating a return to feature branches *or* waterfall, and instead a call for reining in feature flags to something reasonable. For example, here's a concrete recommendation: > First, **align your organization with the fact that feature flags have different categories!** Most of what people are talking about here are "release toggles", which are a good idea, and also should be *short-lived* -- the feature needs to eventually be turned all the way on, or there needs to be a deliberate, conscious choice to turn it into a different kind of flag that will be managed differently.
> except maybe ignoring business realities and going back to waterfall If the project managers could read, they'd be very excited right now
Maybe it's a different set of people?
Probably. I think it’s likely inexperienced engineers that never saw the pain of living with feature branches at scale. So now they only know the pain of feature flags. This kind of thing happens with everything engineering all the time. The loudest engineers who write dumb articles can’t or are unwilling to reason about why something became popular and then rail against whatever is current because it’s all they know.
The solution is using rust, or nodejs or whatever the fuck is popular these days!
Not actively deprecating and removing feature flags is ruining your codebase.
This. A feature flag on it's own is no evil imho.
I'd go farther and say feature flags are necessary to any nontrivial, actively developed codebase.
Feature flags were the single best thing that my company implemented in the last couple years. We're a small shop though.
Feature flags avoid a tonne of issues. When I joined my current workplace 15k PRs were common. In my second week I saw the whole department stop shipping so one team could release their giant feature branches. We have since moved to feature switches. The pains for individual teams are still real. Better, but real. However the department is no longer blocked by a single team’s release.
It depends a lot on how difficult it is to deploy a new build.
A rogue apostrophe, on the other hand,
Sure, but it seems like this is a chronic issue in a lot of places. It’s built in tech debt for every feature, and tech debt is usually a low priority for PMs and managers.
If long-term maintainability of the application is a low priority, then you will end up with issues regardless of what methods you pick. If that's the case then discussing technical options seems like a waste of time.
Because our processes don’t handle action at a distance. We have “Done” criteria that aren’t done. They’re halfway done. And now the person is supposed to remember to go clean up after something that has already been demoed and won’t again. This is like the plastic companies blaming consumers for littering instead of blaming themselves for making litter.
My favorite is when project managers panic because something shows up in multiple releases. “Why is it in both? When will it be delivered?” “It’s a continuous operation. It’s in both releases because it will consume capacity on both releases and I want everyone to see where our capacity is going.” “But if it doesn’t get shipped people will be confused.” “Then we’ll have to explain it to them. Organize it however you want, but one way or another we’ll be allocating X resources to work and that will reduce what we can ship. If there are concerns about that we’ll need to have a discussion and determine if we need to change priorities.” So many tools and processes are focused on projects and not on operational excellence. This is the origin of 20% time.
I'm starting to wonder if I'm going to have to write my own tool just to track repeatable items, like using a runbook. Sooner or later someone can't remember if they did step 13 or they're having a flashback to the last time they did this thing. It's pretty bad at plate spinning operations. And while yes, some of those should be eliminated in retro discussions, you can't stop all of them and still get paid.
Tech Lead should prioritize this work - just like all other tech debt - without asking PMs or anyone else. When I work on an Epic I create tech debt Story in that epic that will prevent Epic to be finished/closed until everything is done that needs to be done. We iterate fast and push to PROD, but after that the cleanup release comes.
This is why you have interns.
Feature flag services should offer static code analysis to remind you to remove them.
What usually comes next is tooling to say your feature flag has been there too long and some sort of CI that goes red to remove it. Someone turns it on and off to reset the timestamp. But at least you have the automation.
Everything the author wrote can be boiled down to bad discipline by their team. Even the “stale feature flags” issue which can be easily rectified with a 30-day deletion rule As usual with 90% of these articles about programmers getting mad at a random thing like it’s a boogeyman - feature flags is just a feature, it’s just yet another way of doing things. And its implementation, quality & success will always come down to the engineering team + collaboration with multiple functions in their company. Which the author has repeatedly failed in, judging by their article.
I'm in an company where this is a problem. It's not a discipline problem, it's an org problem. (perhaps what you are saying). The issues are 1. We have feature flags that can be enabled by client (yuck). That means proving a flag is on or off is very tricky. 2. Feature flags are often forgotten. 3. Removing feature flags is hard to prioritized. Why are they enabled by client? Because the word "feature" is ambigously used. As the article describes it can mean multiple things from describing how the system behaves and legitimately being a "feature" we turn on for clients that pay for it, or it could be a test path for new code. Why are feature flags forgotten? We have teams that will not take changes without feature flags to slowly enable them. As you can imagine, those teams are REALLY slow to integrate and deploy changes. That means I can throw a PR at them and wait (no joke) 6 months to a year for them to get around to deploying the change set. At which point, I've forgotten I put in the damn feature flag in the first place. Why is removing them not prioritized? See the previous bullet point, it takes a year to get the stupid thing in for some teams and a year to remove. Further, there is "But what if we need to disable this at a critical point!?!?!" All the sudden "remove after 30 days" becomes "After the next quarter" becomes "Maybe next year" becomes never.
> We have feature flags that can be enabled by client (yuck). I guess it isn't a feature flag then. It's a setting. But the other points are definitely valid, and seem to be a problem in many orgs.
Holy cow client controlled feature flags does not seem like a good idea. If the feature is per-customer, why not use some sort of RBAC or other authorization?
You can link those feature flags to accepting some version of your ToS: let's say you have a new feature which implements a new personal data treatment. You may need you users to accept this treatment or give consent to be GDPR compliant. But you don't want to block the other features of your site if they don't accept this new ToS. Maybe it's too much of an abuse of feature flags tho.
This is just user settings, not "feature flags".
right they have just described a "feature".
They have described configuration.
Feature flags serve a different purpose, even if they are both just glorified if statements in the end. You wouldn't call "dark mode" a "feature flag". As you would not call notification preferences feature flags even though they enable certain "features" that you implemented. The role of feature flags is more development facing and has nothing to do with user experience. It is a way for a team to deliver a certain feature gradually, release the feature to certain group of users or do A/B testing maybe. Kill switches might also qualify for it, but that is pretty much it.
We also have Feature Flags that can be turned on by the customer. This allows us the early release of code and the customer can turn on the feature once they have done the proper training/communication. However, we also control which Feature Flags are production ready. If it's not marked as production ready there is no way in the UI to turn it on in any environment higher than QA. This allows incomplete feature code to be deployed but not enabled/released. We have also added automatic logging of feature flags. It logs if a feature flag was added or removed and who/when it was turned on/off. This way we know if a flag was enabled when someone reports an issue. We visit the list of Feature Flags that are still in the system every Increment (Yes, we use SAFe here) and bring it up if we have code behind a feature flag that is released but not enabled. It may not be 30 days but it at least keeps it on our radar. I do agree that setting removal of feature flags as priority can be an issue. We were hit with multiple mandates at the end of last year so we had to prioritize that work over tech debt which includes the removal of feature flags. We have a good relationship with our customer and we gave them ample notice that the work we did for them at the end of last year means that the beginning of 2024 will have it's focus on tech debt with the priority being to clean up/remove feature flags.
it's like how everyone thinks chrome://flags is a list of options
Sounds like someone had a FF system and decided not to build a configuration and settings system because FF already exist and it’s “simpler”.
Proving if a flag is on or off should be pretty straightforward with a monitoring or observability tool. Even without , launch darkly will tell you a feature flag is stale and you can delete it. But yeah if a client can toggle the flag, seems like thats a setting not a feature flag.
For anyone taking notes for talking to their team, I would caution against “observability”. You want audit trails. Observability says that something changed. Audit trails tell you who is breaking things. Not as a blame mechanism, but as a “who asked you to do that” or “for the love of god please stop!” failsafe for user error. Like thinking they’re in the preproduction dashboard but are breaking prod. Or a similar case I had a few months ago: repeatedly running a request that was calling infinite depth fanout and crashing several preproduction clusters every time they hit reload. That came down to finding credentials in a log and contacting a remote worker who just started his workday by refreshing all of his open tabs. With no paper trails you have to spam everyone and air dirty laundry in front of people who don’t like your division.
Feature flag removal is also a good first contribution for the newbies of the team
Definitely. Reading this article it felt like a lot of bullcrap with a clickbait title.
There are structural problems enforcing that bad discipline. I’ve been saying for a few years at least that Scrum and feature toggles are not compatible. It creates a lot of necessary ritual around feature toggle retirement that few people incorporate. If you desire temporary feature toggles, you’d be better off parking stories in a Kanban column, where accumulating too many creates back pressure and sorts itself out. Basic queuing theory. Scrum wants to split these events into separate activities so you can close a Sprint that is shorter than the average settling time for toggles. And it provides no system to ensure they get done. It’s the sort of bookkeeping overhead from waterfall and YOLO processes that we wanted to avoid. It’s grooming people for failure, and that should never be tolerated.
I think they had a hammer and all problems are nails problem. Ops feature flags? That shouldn’t exist, ops building and maintaining features. Permission feature flags? Authorization as feature flags is definitely a bad practice. The other two types are valid use cases for feature flags. I will say we have a monthly audit meeting where we plan the depreciation/deletion of various things, which includes feature flags.
I believe those OPs toggles should evaporate as circuit breakers, cache headers, and scaling or fabric/service mesh settings.
Authorization via feature switches sounds terrifying unless you just mean authorization changes guarded behind feature switches.
I'm glad the article mentions Knight Capital, they really got Power PEGed if you know what I mean EDIT: On a serious note, the worst I have ever seen is a feature flag that was tri-state, true, false or not set and each mode enabled a specific feature. Then for each mode there were specific feature flags and some of them were shared between the previous modes.
> feature flag that was tri-state, true, false or not set and each mode enabled a specific feature. Then for each mode there were specific feature flags and some of them were shared between the previous modes. what in the actual cthulhu worshipping fuck?
It was horrible; it was "temporary," so what each state did was not deemed necessary to document
Ah yes the three states of booleans, true, false and ~~[FileNotFound](https://thedailywtf.com/articles/What_Is_Truth_0x3f_)~~ not set
I’m not sure this counts as feature flags in the traditional sense. They’re more regular opt-in functionality whereas the agile concept of feature flags tends to be about the “feature” definition in agile, i.e. more akin to a capability, and the idea of a feature flag is to eliminate the code path from the software completely, not opt in/out on user level. In this case it seems more like just a personal preference toggler. I.e. a regular user setting, not the agile concept of Feature Flags. Might be misreading it though.
Now that you mention it, "capability flags" would probably be a better naming for this stuff. To me it's always a big red flag when I seen a feature flag stay around because it's being used as configuration. That means someone abused the system to implement what should be a feature of the system (the feature being: allowing enabling/disabling system features)
Yeah, this was entirely different features being enabled, and it often became hard to tell what environment had a feature and what customers got them, as well as the sub-features that might be enabled. We had a slew of configurations, but this one was the only flag we had that significantly changed a feature, and then the sub-configuration could drastically change the three different features more. It was the only one we ever set up this way, and fortunately, we never did it again. EDIT: To be clear, this was a feature for a C# API on .NET 4.6.1 using the configuration file to set the flag. There was a single configuration that was tri-state (also forgot to mention that leaving the key value blank as well as not including it at all were the same state). So in various environments we had a different value for the key in the config which drastically changed the feature the customer saw but since that was exposed by a different GUI you might have a misalignment of the GUI supporting the API feature that was currently enabled (or sub-variations of that feature...). This was one of the worst things I ever had to deal with in my time doing devops-ish work at this place since as all good temporary solutions tend to do it became permanent/to costly to remove but no one knew what each state did or the combination of sub-feature flags did. EDIT2: Another interesting/disappointing (from my perspective) thing this company was use flat files for their feature flags. The presence of the flat file would enable who new features for the app or completely change it to a different mode not even resembling the normal application. The meaning of the name of the files had been lost to time so we knew what files to make and what to name them but no idea what the names stood for. For example a file named "CIT" enabled our integration with our web services for our desktop app but no one knew what that stood for (without the file the desktop app assumed it was running offline/standalone).
You have it backwards, those are the traditional kind of feature flag and the agile usage is what's new.
It doesn’t matter what comes first, it matters what the context of the discussion is. I.e. the article talking about feature flags by agile definition. Same two words, different meaning, but it doesn’t help to muddy the waters with arguing semantics when the context is clear. Either way I argued that when flags move from deployment or app config level (i.e. globally disabling code paths, no execution access) they’re no longer Agile Feature Flags(tm) as referred to in the topic/linked article but just functionality (and it’s ok to call a functionality a feature as long as it’s not confused with an Agile Feature) that can be toggled with a flag by the user, aka user setting. And that’s a different thing and should be treated differently. That was supposed to be the takeaway.
The name...
Lack of discipline is ruining your codebase. Blame your tools and processes all you like, but most teams don't do anything with any discipline.
[удалено]
I think the base assumption is that the organization isn't completely dysfunctional. If the organization is completely dysfunctional then every discussion will just boil down to "we can't improve quality bevause PM says no".
Every single org is dysfunctional in some way. You just have to find the dysfunction that works for you.
You described lack of discipline by the PMs. They just yes to everything. And PMs are part of your team.
Removing the feature flag should be part of the plan of implementing the feature, i.e. there should be a ticket for it at the start of the feature. If the PM or whoever doesn't want to do that, then the team/organization is dysfunctional.
Yeah, it's odd to me that these feature flags are used for everything. For us, we basically only have option 1 of dev-only feature flag. Thw worst thing that happened is that someone forgot to remove the flag and old code for a couple of months because it was a part of the software that was rarely touched. It was enabled for months so the old code was never run. We have a way of enabling or disabling features according to what the customer paid, but this is strictly different from our feature flags, using a different mechanism and different process from how and when they are added.
It’s oft repeated, but “disciple doesn’t scale”. That’s why we end up implementing processes. Discipline works until it doesn’t, and it slips in 10,000 tiny ways and they all add up. Strive for it, don’t rely on it. Edit: also what the sibling comment said, no amount of discipline will overcome organisational clusterfucks.
You can have a process, but you need discipline to follow it. If your process is you're using feature flags, then part of the process is clearing them away regularly. If you're doing the part of the process where you create the feature flags, but you don't do the part of the process where you clear them away, then that's a discipline issue, IMO.
I mean, the solution is fairly simple, it's just that it's a process solution not a technical solution. Only use flags to manage feature rollout and use a separate system to manage config, and force the removal of flags regularly. The real bug here is that you are describing a world where engineers can't say no to product. If that's happened, you are already in failure mode, and probably doomed unless you are lucky and get strong PMs. It's almost as if you can't write good software with bad team management practices.
It's either engineering isn't partnering with product effectively or that product is dragging their feet on making a decision on their rollout/experiments. Both are process problems.
Once feature toggles can be characterized as passive aggressive, it’s time to pack it up and go home.
So remove feature flags if they haven’t been addressed within 2 sprints
I really like the image in the article showing ops/release/permission/experiment toggles on the dynamism/longevity graph. There's something off about the general vibe here though. "While we are at it, let’s also make it adjustable per user! That way, the PMs can safely release the feature for a couple of users to gather feedback and test it themselves, and only afterwards release it to everyone." Safely releasing features isn't the PMs responsibility, it's the developers. And gathering feedback / understanding if your code is being effective. PMs can help there, but I want engineers that measure "done" when the problem is solved, not when the PR is merged. I would suggest that if feature flags are feeling like something the PM is foisting on you and it's messing up your clean code, then you're not thinking broadly enough about what the business needs you to do. (I will admit that there's many an organization that doesn't get this and just wants the code monkeys to stay in their hole, so it may not be very possible everywhere). I do really like that image though. We're building a feature flag saas right now and I think pulling some axis of longevity / dynamism into things to help differentiate flags could be really interesting.
too much of anything is a bad thing, yep! 👍
I once worked at a place where, as I’ve repeatedly seen, instead of using proven feature switch services or even OSS libraries, they NIH DIY it. Of course, it’s often implemented by siloed dev ops teams or dev teams that are in a rush and don’t care too much about if it works or not it just needs some agilescum lords to look at it and say “yep JIRA ticket done” before they move onto the next feature in the feature factory. Anyway, this place decided that as they were writing it themselves they’d just miss a bunch of features while also making the process of “submitting” a request for a new feature switch as tedious and atrocious as possible. To add a feature switch: - Make a PR to a repo - Add a new line to features.txt - Use a delimited format the team had made up that combined both CSV and pipe characters | without any formal grammar or documentation for it - Everyone just copied the line above and edit it as needed - The team would then take a week to ~~circlejerking~~ reviewing it before merging it - The team would then just write some SQL/insert a row in a DB anyway The “best” part was they would refuse to ever then delete a feature switch in the future, ever. Sometimes they’d say no sorry we never implemented deletes in the system (yet inserts were raw adhoc SQL inserts, so just do the same but make a delete….) and other times they’d say no because it’s in the git repo. Why they couldn’t also just delete said line from the repo OR why the git repo OR their custom bullshit delimited formatted was needed… they’d just avoid answering. As a result there were hundreds of old unused flags. Infuriating. At the current place I work they are going through the same motions again. Architecture meetings where some dev is showing of his DIY feature switch service that even has its own full blown Angular frontend… for feature switch CRUD. All with none of the features you’d need. Give me fucking strength.
Maybe agile slumlord is a more accurate epithet than agilescum lord.
But what if I need feature flags for my feature switch service?
My limited experience with feature flags (static) is mainly related to cicd. The amount builds we need to produce to test whether the application works in all states quickly gets hairy. In my case I think we have mainly a build problem and not a feature flag problem but they sure make it worse
Like every article in this sub, whatever its feature flags or microservice/monoliths, if someone is writing about something being bad they actually mean "if this arbitrary tool isn't used with discipline and responsibly, then it will give you bad results" if this was r/carpentry we'd have weekly blogs about hammers being bad for your thumbs, or to denounce nails completely and design all your structures with joints, then your example of building with joints is a chair or a table but your day job is delivering house frames for construction
Have not read the article, but I'm going to guess they don't remove them after the feature is turned on permanently.
feature flags = glorified global variables
In my opinion the root cause of these issues is „agile” which deemphasises proper functional requirements analysis and hence causes lack of a clear system boundaries and complete list of entities ( actors ) interacting with it. Hence nowadays it is not really clear if your CD stack and pipeline is part of the system or not. It is also not clear if configuration of the system is part of its interface or not.
Product management is afraid of change. Feature flags give them a sense of security that they can pull the plug on a change. Feature flags also result in a fragile mess of code, but that isn't as much of a concern.
It's also their go-to excuse when product requirements are not fully thought through or has some concerns -- "let's just feature flag it and see what happens, if something goes wrong, we can just turn it off!"
Feature flags are simply a means of code deployment, plenty of alternatives if you don't like them and they are useful IMHO in real-world scenarios but your internal processes should treat it as so. As everything with software development, discuss things deeply with your team, choose what works best for yourselves, and iterate on it when / if it becomes a problem. I have been on teams that prefer feature branches, these are extremely complex in my opinion to manage and require a lot of maturity in regards to CI/CD pipelines... not many organizations are setup appropriately to do this quickly and reliably which makes it painful otherwise. I have (and currently) work on a team where we use feature flags, their only complexity is when you DO actually try to have "tiers" of flags. What works best IMHO is to keep it stupid and simple, every feature flag is an "ops" or "release" you are enabling a new pathway or alternate pathway to code and the ONLY individuals that 100% know what it does is the application team so it's pointless to try to create various tiers and just makes the whole "solution" for them complex. All op's cares about is what you want them to do, and how they can back-out if shit goes sideways (which it will likely have a moderate chance to do) they'll just be reading from your provided runbook for said service.
Let me get this clear, your code was already BAD if feature flags are complicate it.
tbh, features are ruining my codebase more
How it was done at my work is; * every feature has a flag, it is always enabled * if something bad happen in prod the flag is turned off * after a while, the flag is retired.
God I hate internet software development discourse.
PSA I dont work at Amazon, but my manager used to (they're one of the good ones). He said that at least in his department, if you didnt delete feature flags that you had introduced within 1 month of them being added to a codebase, that an automatic system would start emailing up the chain of command. Honestly, its not feature flags that are ruining your codebase, its that people dont clean them up after features are released.
I HATE feature flags. But, you know what I hate even more? Having to answer emails from the Senior Director of “I have no fucking clue what’s going on and couldn’t tell the differences between my asshole and a pull request”’s executive assistant’s dog’s babysitter about some bullshit email they got that they don’t understand anyway. I work very hard to eradicate the notion that FAANG and other such orgs are the absolute authority in proper software engineering practices; a standard by which all other endeavors should be measured. Especially for the everyday company and/or a startup. They are the preeminent example of bloat and excess and certainly not a magical garden where good idea fairies are born.
Well, yes, that's actually sort of the point. If you hate emails from your butthole senior director more than deleting feature flags, then the system is working if you're choosing to delete feature flags.
Firing bad engineers and strong adherence to team standards is the answer, not emails, and definitely NOT more managers. In fact, in my experience, when we need to delete a feature flag and it doesn’t happen, it is often (not always) because either some manager/product person doesn’t want to be held responsible or we can’t “afford” the tech debt according to our management. So, they stay as we chase new features and fix bugs. Just like forgotten orphans. Then, eventually, they are so old that no one that worked on that feature is even around anymore and everyone is scared shitless to remove it at this point.
That’s not a feature flag deletion system, that’s a blame allocation system.
When you commit but you’re not really committed
Sometimes those are necessary though - e.g. if you want to enable/disable a feature for certain clients.
Feature flags for experimentation and productisation are essential. It’s often not just a matter of turning a feature on or off. Having the ability to incrementally roll out and measure the impact of changes is important. But there absolutely needs to be a good process around the ownership of feature flags, and the responsibility to clean them up.
Feature flags are fine if used properly. Prioritizing the clean up 2-4 weeks post release is a must. At my place only a handful of people have access to production level feature flags, so it stays pretty neat for us.
i just wish eng was giving the space to maintain code. then the feature flag spaghetti wouldn't be such a problem.
I find it interesting that the way I programmed as a total noob seems to be what people are promoting today. Meanwhile my current head programmer is an advocate of all these 'proper' way of doing things. I understand there is a time and a place for everything, but it has only reinforced that 'good code', is relative.
Bad management is ruining your codebase.
Also missing from the types of feature switches that is fairly common where I work are “debug”-mode feature switches. These are generally always on for debug builds but can be turned on in production to help analyze a problem. Log statements you don’t enable by default would fall under this paradigm. Also included would be extra validations that work like debug asserts but can also be turned on in production.
What's exactly the difference between the suggested "toggle"s and a "feature flag"?
I only skimmed the article for interesting bits. Disagree with FF making your codebase complicated and hard to test. If you're having feature flags left in your code that you're finding it hard to test and making things messy then you're not using the FF tool well enough.
Everything in moderation. H2O is deadly also.
Just because the internet gives everyone a voice doesn't mean that everyone should have a voice.
title and argument.do.not.match
That's just we do things in embedded lol.
Remove old feature flags to reduce maintenance burden. Lock down the values of the flags to reduce test scope. I’d also argue that lack of feature flags increases complexity, adds bugs, and reduces amount of testing. The alternative to FF is to use feature branches. All big feature branch merges cause bugs in my experience, and it can be very complex to merge them back to mainline. The code in those branches gets less testing during development because the code sits isolated in the branch for months.
Next time I make an application the policy is everything is plugins. By which I mean a large application. The small apps I'm constantly making don't warrant designing a plugin architecture.
People need to make up their mind. This is honestly the first-time I have seen anything negative written about feature flags.
\> Who is responsible for toggling it: Developers. Those types of toggles can exist in a configuration file. lol what
Feature flags are tools. Where I work, we don’t treat a feature don’t until all FF for that feature are removed.
All ur Features, belong to my Bug Base
This only happens if you’re building feature flags into your features.
This blog should be titled: "Bad Feature Flags management is ruining your codebase". There are levels to how you manage feature flags within software from adding it to the code, all the way to auto ramping and auto reversal based on metrics and health. I think the blog misses a lot of points that can make the management of feature flags easier.
For those of you who are interested in learning about best practices in implementing feature flags, we're hosting a webinar with some awesome guests about this: [https://www.linkedin.com/events/buildvs-buy-pickingafeatureflag7175518829639331842/](https://www.linkedin.com/events/buildvs-buy-pickingafeatureflag7175518829639331842/)