As always when I write about “anti patterns”, or “things not to do” – I’m speaking from experience.
In the case of over-engineering, not only did I use to do this “bad thing” – I still sometimes do it today – knowing full well it’s a “bad thing” to do!
For me, this proves a tough habit to break.
This is partly because I wasn’t actually clear, until recently, what “over-engineering” actually is.
I used to think that “over-engineering” is over-application of “engineering”. Like:
- Creating a 7-layer architecture for a CRUD app
- Using redux for a website with less than 5 pages
- Using kubernetes when you’re not google 😉
But that’s not over-engineering. that’s just bad / overly-complex / resume-driven engineering.
Over-engineering, as I now understand it, is:
Building functionality that is not required.
(yes, “required” can be a bit blurry. For simplicity’s sake, we can define “required” as “appears on the work ticket we’re currently working on”)
On the face of it, this looks pretty simple. You’d have to be pretty dumb to work on something that nobody’s asked you to do. Right?
Well, what about “future proofing” and over-generalizing? Have you ever done those? I have do!
These are, actually, cases of building functionality that is not required.
As usual, XKCD explains it best:
How many times have you implemented a system to support any arbitrary condiment, when the customer just wanted some damn salt?
Some other examples of over-engineering, from personal experience:
- Recently I added a new string field to a data model. We just had to read it from a form, and later show it in the UI.
A colleague suggested we write code to normalize the values in that field (e.g. downcase, trim whitespaces), as well as create an index in the DB for it. This is in case we’d need to filter or sort reports by that field in the future. - I was tasked with building a very simple survey tool. All the questions asked would be “yes/no” questions.
I implemented a system that can process any text answer. - When calling out to a 3rd-party API, we wanted to retry 2 times when an error occurs, in case the error is transient.
We’ve implemented a configurable system that retries X number of times. - When a user submits an identifier of an OWASP top security risk (e.g.
A03), we needed to show some information from OWASP about that risk.
I was already planning a real-time API client or web-scraper, and how we can cache data even between requests to improve performance and resource utilization. That way we’ll always have up-to-date data in real time.
My colleague just scraped the OWASP website into a JSON file, and set himself a reminder for next year to check the new top 10 list.
Also, have another look at the examples of “bad engineering” I gave above. In some contexts, they too can be considered over-engineering:
Building extra application layers “in case” we need to add more logic. Using kubernetes “in case” we need to handle web-scale load. etc.
Why is this a problem?
Some of the above “over-engineering” is not very complicated or difficult to do. Downcasing some strings, or reading from a configuration file. Is it really such a problem?
Keep in mind, though, that the cost of initial implementation is not the largest cost associated with writing code.
Any code that’s written requires maintenance.
Every developer looking at that code for the first time needs to figure out what it does, and why. How confusing is it when the answer to “why?” is “no reason”!
Every existing functionality needs to be preserved. So from now until forever, we need to be careful to not break it as we change the code around it. We need to regression-test it every time we make changes.
furthermore – when we guess at additional functionality, we guess that it may be required along some axis X.
And we write code based on some assumptions about X.
However, the actual functionality in the future may be along axis Y, which is different to X.
Now we’ve painted ourselves into a corner, designing our software to handle change in the wrong direction.
(For example – we designed a system to configure the number of retries for an API call.
But what if, in the future, we still want to only retry twice, BUT, we need to control the amount of time between retries? The “extra” code to check how many times to retry may make the implementation of configurable wait times more difficult)
Why do we do it?
Hopefully I’ve convinced you (or, you already knew) that implementing unwanted functionality is a bad idea.
So why would developers who are otherwise smart, skilled and reasonable, engage in over-engineering? Making their code harder to work with, in order to build functionality that nobody wants?
Are they crazy?
No. Quite the opposite – we over-engineer because we’re smart.
Here’s a very logical thought process where the logical conclusion is over-engineering:
- It’s difficult to understand and / or to change this piece of code. AND / OR
- It’s difficult / time consuming to validate that this piece of code is working as intended.
- I currently have to change this piece of code, and validate that it works as intended (e.g. because I’m extending this functionality).
- This is a difficult / time consuming process due to the above.
- If I have to do this again in the future, this will result in more difficulty and time wasted.
Conclusion: As long as I’m here, I might as well make some further changes that may be needed in the future, even if they’re not needed now.
This will save the overhead of having to understand and / or validate this code again in the future.
Statistically, this seems like a sound conclusion.
Let’s say that I’m investing a further 1 hour of work on those not (yet) necessary features.
And let’s say that this 1 hour now can save me 3 hours in the future, if my guess about the future requirements is correct.
In that case, even if my guesses are only 34% correct, I still come out ahead.
I like dem odds!
However, if we look a bit closer, the above logic is flawed.
Specifically, the assumptions that we begin with:
The code is difficult to understand and / or change and / or validate.
But this is not due to some act or god, or a law of nature.
The code is difficult to understand because we wrote it this way.
It’s difficult to validate because we wrote it this way (or didn’t write good automated tests).
That’s why I claim that over-engineering is a cry for help:
If we start from the result (“we are over-engineering”) we can work our way back to the reason (“the code is difficult to work with”).
If we do that, we can understand the reason behind the need to over-engineer. Then we can hopefully deal with it, rather than doing the poor-person’s optimization of over-engineering.
Making our code safer and easier to change will make over-engineering unnecessary,
which will help to keep our code simpler,
which means it’ll be safer and easier to change,
which will make over-engineering unnecessary…
rinse and repeat.
On the other hand:
Over-engineering means that our code will be more complex (because we’re adding extra code and functionality),
which means it will be harder to understand and to change,
which will create an incentive to over-engineer,
which will make our code more complex…
rinse and repeat.
Which of these cycles would you rather be on?
P.S.
I don’t want to ignore the fact that writing simple, easy-to-understand, well-tested code is hard. It’s a skill that i still haven’t mastered, 15 years in.
But if we don’t face this difficult challenge now, we’ll end facing the impossible challenge of changing an over-engineered mess.
The Useless Dev blog 
While I understand your point, the “YAGNI” approach is sophomoric and should be avoided.
What is missing is the application of value engineering to adding some functionality not specified in the requirements. If a net positive value can be defined for adding it, then it should be added. If not, just put the concept of that functionality into a “future additions” document and let it go.
I have seen too many applications where substantial code rewrites had to be done in later versions because the software engineer didn’t think ahead, apply value engineering, and code something not specifically required that would have eliminated that rewrite.
LikeLike
It all depends on whether you can communicate with the wider product system personnel.
The over-engineering identified is typically a ‘great thought’ that hasn’t become an implementable product design feature – one that the wider product team thinks is beneficial. You/they/we have made a *guess* about the usefulness, rather than talking to stakeholders and decision makers about the issues you foresee.
It’s about talking, rather than coding.
LikeLike
Interesting. In the hardware world, over-engineered means that you design for extra headroom like using a higher voltage or higher current part than you need “just in case” other parameters can get this treatment as well, i.e. faster, quieter, cooler etc. Also colloquially called “bullet proofing” something.
LikeLike
At least your examples 2 and 3 are misleading, and probably wrong – i.e., they are not over-engineering.
You start by saying, in tiny print:
“yes, “required” can be a bit blurry. For simplicity’s sake, we can define “required” as “appears on the work ticket we’re currently working on””.
But this is wrong. “required” means “the solution to the problem requires it”. The problem with this is of course that we humans (and especially those who wrote your work ticket) often do not (or at least not yet) know what the solution really requires. Re your examples:
– “All the questions asked would be “yes/no” questions.” Experience has shown again and again that even for “yes/no questions”, values like “n/a” (not applicable) or “other (specify)” are needed at some point – making enums and strings useful answers. If this isn’t a single-use fun throw-away project, I’d go another mile with the customer/requester to make sure that never, ever, such additional values will come up (and even then, would not believe it, having written software for 40 years now).
– That you “wanted to retry 2 times when an error occurs” is almost certainly not a requirement – it is a guess, or hunch, with maybe some base in experiences. In that case, it DOES make sense to have a flexible solution to adapt this value later, maybe even in production. (It would be a requirement if it were written in some standard, or some other quite hard document to that effect. Your write-up doesn’t sound so).
So, yes, you are right, one should look out for over-engineering. But one should also look out for under-specifying, in this complex and dynamic world.
H.M.
LikeLike
Before storing user-entered text in a DB I really think that trimming leading and trailing space is a great idea that you should do Right Away. The support calls trying to debug why a query for ‘foo’ isn’t matching the ‘foo ‘ they entered when they signed up will be excruciatingly painful. That, to me, is not an example of over-engineering.
More generally, everyone can agree that over-engineering is bad practice. But under-engineering is a thing too, and the consequences can hurt. One of the ‘lifetime to master’ aspects of software engineering is working out whether some proposal is under- , over- or just-right-engineering.
LikeLike