When people think about software engineering the first thing they think about is programming. This is a reasonable thought. But programming is really just the beginning of what it means to understand and be great at software engineering.
Engineering at its highest abstraction is about building and maintaining systems to solve problems. Systems rarely comprise a single program. Systems are constructed by connecting components together and engineers make some guarantees about how these components will work and interact in the face of various loads.
Software engineering involves requirements, architecture, design, algorithms, denoising, development, abstracting, testing (of which there are many forms), debugging, benchmarking, optimization, scaling, configuration, tooling, security, maintenance, refactoring, and workflow processes. (Let’s leave relationships like team members, bosses and customers out of this discussion for now)
In all aspects of engineering, complexity will arise.
Accidental complexity is caused not by the inherent nature of the problem being solved – essential complexity – but by your/your team’s approach to solving the problem.
Accidental complexity can be created by your choice of tool, or algorithm, or a poorly designed abstraction. Your approach may not be inherently bad or invalid, but for the purposes of the problem being solved it can introduce unwarranted complexity.
Here are some strategies for battling accidental complexity.
Reuse code. Integrating code that has already been written and tested and run in production is a very good thing. Production quality code is more apt to have been tested to handle various edge cases. Re-writing code from scratch, while sometimes necessary, is inherently prone to errors. The new code base is likely to start out simple but will accrue new forms of complexity over time that another piece of code properly addressed. Joel Spolsky does a fine job talking about code reuse here and here.
Automation. Tools that allow any type automation should be incorporated into the team’s workflow as soon as possible. Automation can be applied to testing (junit, pyunit), development builds (maven, paver), configuration management (chef, puppet), application testing (selenium, cucumber), continuous integration (jenkins) and pretty much all other functions. Automation brings consistency across the team. Consistency maintains simplicity — each team member doesn’t have his or her own way of doing things — and automation frees engineers to focus on problems and not on tooling.
Good abstractions. If a piece of code can’t safely be reused or automated we’re now likely getting to the place where custom code is required to solve the problem. This, however, doesn’t mean you can’t build towards future code reuse or automation. Good abstractions enable these things. Good abstractions empower you to build a system using architectural patterns such as layering and builder patterns. Good abstractions mean appropriate interfaces and APIs to your components and being able reuse these abstractions across your stack where applicable. Components built using SOA reduce complexity by increasing composability: they can be debugged and refactored without being muddied by the logic of other components, they can be swapped out and replaced with a shinier version, moved upstream to a different part of your data pipeline, or re-used as part of a new backend service. An interface is a contract between the component and the rest of the world. Abide by it and your system as a whole benefits from increased maintainability and simplicity.
When you have a toolbox everything doesn’t look like a nail. This one is simple — Use the right tool for the job. As mentioned above consistency is important. But taking shortcuts or trying to retrofit a tool for a task it’s not built for will only increase complexity and technical debt of the team. Continuously evaluate your toolbox and question whether there is a new utility or library that’s better for the job.
Code reviews. Code reviews are cultural. They require that all teammates come committed to the table with respect, open-mindedness, and pragmatism. A code review is not meant to solve every bug but it is helpful to 1) shed light on best practices and 2) increase shared knowledge among the group about the code base. Both these things are very effective ways to reduce accidental complexity. Code reviews foster collaboration and help mitigate the impact of coding in isolation. Knowing what each teammate is working on will help each engineer think more holistically about their code and determine whether the class they are working on should be part of a more generic package that all engineers would find useful.
There is no magic bullet. Complexity is very easy to build and simplicity is very hard. It takes discipline and committed thinking and execution on the team’s part to engineer better systems and focus on solving real problems.