I have a growing opinion that I think is highly controversial in some circles: We have become too obsessed with objects, “object-oriented design”, and syntax of data structures. We are surrounded by objects-run-amok and syntax everywhere: Design Patterns, Domain-Driven Design, even JSON. We think objects and syntax make our world better. But what if they are actually making our software more coupled, our code another step removed from the data, our systems harder to integrate, and incurring all sorts of versioning problems along the way?
“I invented the term object-oriented, and I can tell you that C++ wasn’t what I had in mind”
Common software problems
Similar to speaking languages, programming languages shape the way we view the software world. If all you have is a hammer, everything starts looking like a nail. If you’ve only worked with object-oriented languages before, you will naturally think of objects first when writing code. More and more students are taught this model as they often are using a language like Java or C# as their first programming language in schools. While we learn this way of thinking, we rarely take the time to step back and reflect whether it’s actually making our lives better or not.
There are three problem areas that I see cropping up all the time regardless of the technology being used:
- System integration is hard
- Versioning is hard
- Composition/Decomposition is hard
In many enterprises, I’ve seen system integration becoming more and more vital. Whether it’s a move to Microservices, Workflow systems, Data Science, or something else entirely, we are constantly needing to integrate multiple systems together. These systems are often not created with the intention of easy integration with other systems. So we live in a world of dealing with data in all sorts of different shapes or densities, missing semantics, proprietary communication protocols, business processing rules, etc.
Versioning is fraught with challenges. It’s not a simple matter of “just use semantic versioning and you are good-to-go.” Semantic versioning doesn’t really address our versioning challenges. How many times have you been broken by a minor change in a new library version? How many developers think of bug backwards-compatibility when versioning software? Clients may have come to rely on the behavior of that bug, so fixing the bug could be in fact a breaking change itself. And what do you do with intentional breaking changes? Simply increasing the major release number doesn’t make anybody’s life easier. It’s telling people “you might as well change a bunch of your software because it can no longer be guaranteed to work now”. At that point, you might as well just change the name of the project or library.
Finally, decomposing a problem into concrete processing steps or modular structures is incredibly difficult. There are some good papers and books out there talking about this: Big ball of mud, On the criteria to be used in decomposing systems into modules, Society of Mind by Marvin Minsky, and Niklaus Wirth touches on it in his paper A plea for lean software for starters. Finding the right data structures is an iterative process and one that needs to be focused on learning.
The thing is, Object-Oriented Programming (or how many folks think of OOP) doesn’t actually address these problems. In fact, it can make solving these problems more difficult in many ways.
I’m sorry that I long ago coined the term “objects” for this topic because it gets many people to focus on the lesser idea. The big idea is “messaging”… The Japanese have a small word - ma - for “that which is in between” - perhaps the nearest English equivalent is “interstitial”. The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be.
Alan Kay (AlanKayOnMessaging)
This interstitial stuff, this “ma”, is everywhere in our programs and it’s usually implicit and hard to reason about. We often don’t see it in the first place. It is effectively a contract that all our code has to abide by. It’s the contract that dictates exactly how callers interact with a function (the order and structure of function parameters and the function’s return value). It’s the contract that dictates how functions are composed together (how the return value of one function is automatically “passed on” as a parameter to the second function). It’s the contract on how you communicate with an object (its properties and methods). It’s the contract on what order functions need to be called in (protocols, an object’s methods that have to be called in a certain order). It’s the contract of how modules interact with each other (what functions, objects, etc. does a module have that can be used externally by a different module). It’s the contract of how different services communicate with each other (binary protocol, web service, file sharing).
I find that this stuff isn’t really deeply thought about much when a system is being developed. We are so used to thinking in objects that we end up using objects everywhere and end up making our lives more difficult. We let the types take care of it all. But these interstitial connections are one of the large factors that make building software systems difficult. While we may capture the interstitial syntax through the shape of our objects and data structures, we very rarely (if ever) capture the more important interstitial semantics that the system is built upon. The semantics effectively grow out as an implicit by-product of how we wire our code together. They aren’t captured as first-class citizens.
Increased coupling and brittle software
At the end of the day, should I really care about the shape of the individual data structure coming into a procedure, or do I care more about being able to retrieve a specific piece of data that has the semantics I wish. If I want to get the author’s first name, should it matter whether it’s available as
obj.Author.Name.First? I don’t care about the syntax of getting the author’s first name, I want to use the semantics to ask for the author’s first name. (Yes, this is very similar to the thinking of Linked Data and the RDF/RDFs space in general but applied in the small; it’s also somewhat intertwined with Clojure and the emphasis on Maps as a common “communication” data structure).
We know that Premature Generalization is A Bad Thing. But with our focus on object-design and object-first thinking when using OOP languages, we are often generalizing from the get-go. You can find this in run-amok class hierarchies in many projects. The thing we fail to realize is that we are not actually abstracting away, we are making things more and more concrete. As Rich Hickey put it, these are not “Abstractions”, they are “Concretions”. Set or Map are abstractions. PriorityQueue is an abstraction. LinkedList is an abstraction. Are Employee, EmployeeServiceFactory, or EmployeeRepository really abstractions? Our concretions are another way that our code becomes dependent and coupled.
Well, certainly that’s what we have interfaces for, right? There’s the Dependency Inversion Principle and all that jazz. But now we’ve introduced more interstitial typing into our code that makes it still more brittle. What if you want to change or enhance the interface? Now there’s another type too that is in our floating universe of types we have to remember and work with. Remember, none of these types capture the semantics of communication either, only the syntax. The coupling is still there, it’s merely hidden behind an added layer of indirection.
With so much of our focus on the modeling of the containers themselves, we often lose perspective of the data itself. Data is something that’s hidden away behind objects as internal state in an OOP world. Our domain is no longer modeled as understanding the data in our system, it’s modeled as objects that control and manage the state in the system. We have effectively hidden data behind an added layer of indirection. And because the objects only dictate the syntax and not the semantics, there is no way to tell that
Author.FirstName actually refers to the exact same concept as
Employee.FirstName. The data that represents a first name has now been namespaced and hidden away by the objects with no easy way for the system to tell that they mean the same thing.
All of this limits the reusability we can get out of our code. If
Employee aren’t directly related, what if we have a function that works with people’s names that we want to re-use? In a statically-typed system, we can’t simply pass in the author or employee objects to the same function unless they have a common class ancestor like
Person they both derive from. And what if the function was built to work with strings instead? These problems can be solved of course, but we aren’t encouraged to fall into the Pit of Success. We have to spend energy and thought working against the system to make our lives better.
What if we want to change the structure of our objects? Well if we do that, we need to change all the tests, any code that uses those objects, etc. Because our entire system is built on the House of Cards that syntax can turn into, everything falls down if anything moves. This kind of rippling change across our code is a symptom of increased coupling and represents that we are building fairly brittle software.
This is an area that I’m a fan of idiomatic Clojure code. The reliance on a Map as a foundational data structure that functions operate upon makes this sharing a lot more possible. I still wish it was semantically driven though instead of syntax driven. But using namespace-qualified keywords/symbols certainly helps in that regard.
At the end of the day, a type defines a contract that other code needs to adhere to. What are the impacts on versioning of software at this level? We don’t version data structures really today. If you are introducing a new member in an object or changing an existing one, there’s no version number to increase. You could place the object in a module and version the module itself, but it’s not very granular and forces us to do something not otherwise necessary for versioning purposes.
Again, we use languages that are reliant on the syntax of a data structure (it’s shape). We spend much less time focusing on the data being passed around the system and the semantics of that data. Some languages even define the layout in memory of an object (it’s “v-table”) and metadata is used when code is compiled to know what slot of the object to reference. What if the order changes though? You’ve now broken your clients. If we took a different path… what if every property were accessible via a URI? Would it matter where it was in a data structure if you could just reference it via URI?
We’ve become complacent. Versioning at the module level and changes that are easy to make to an object thanks to OOP languages and tools for refactoring has us thinking about the impact of our changes much less frequently than we should. If we were more concerned about the underlying data and functions that operated on the semantics of the data, how much would versioning be less of a problem? In a “magic world”, where everything operated on a Map let’s say, and all things were looked up in the map via URIs, how much would versioning apply? If the semantics of a thing changes, usually it means it is a different thing altogether. So versioning becomes more a matter of naming and URI identification rather than versioning per se.
Poor software composition
“It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures.”
Being surrounded by objects everywhere, we are often faced with a situation where there are just a handful of functions that operate on each object type and that reusing a function across different types is incredibly difficult (and often done via inheritance or a different form of polymorphism).
We usually end up finding ourselves in a situation where we are inundated with types and the data they house is extremely hard to share or re-use. In Functional Programming, we build up from smaller pieces using function composition. But what is the equivalent in the Object-Oriented world? Well, we have Composition-Over-Inheritance. But now the high-order bit that is flipped for composition is the data structure itself, not the functionality. What if you wanted to use methods from a different object? Unless you come from a language supporting mix-ins, you’re going to have to go out of your way to make it possible.
What we’re really doing is composing syntax on top of syntax on top of syntax. We make ourselves live in a world of exponential complexity growth by default. 10 data structures with 10 functions each in the OOP world isn’t just 100 things to deal with. We now have to think about all the ways they are interconnected as well, especially once we start composing them together in the OOP way. Our interstitial space we have to worry about has grown a lot larger than we would have otherwise had to deal with. And what captures that complexity of the interstitial space? Nothing. It’s implicit and transient.
And whether you are discussing OOP or FP, you are still in a world where the way we connect systems together in the large is different than the way we build programs in the small. There is a context shift we have to mentally make every time we jump up or down the abstraction ladder. The way most programs are built in the small simply doesn’t scale up to the large. We need to find better ways to compose larger systems together from smaller.
Am I saying that Object-Oriented Programming is bad or inadequate?
No, OOP is not bad or inadequate. But sadly, I think we focus on the wrong designs and the wrong things. It even encourages us to do Bad Things by default in my opinion. I do happen to think that developers could benefit by exposing themselves to other ways of solving software problems, like functional programming, logic programming, immutability and message-passing, and more.
That sadly seems to continue to be a rare thing in the industry as a whole though.
Does Functional Programming solve these problems?
While functional programming and lambda-based thinking helps with some of these problems, it’s not true across the board. It’s a step closer as it encourages thinking about the data instead of the containers. But we still build systems at the micro-level in ways that are different from how we compose things together at the macro-level. And even with functional programming, there is a tendency to design in terms of Algebraic Data Types to an extreme degree. They’re just ever-changing and implicit based on function return types and function parameter types. Even in a dynamically-typed language, this distinction still exists as you are still dependent on the shape of the type.
Maybe it’s because of toying around with LISP lately, but I like the the focus on the data and programs being data themselves. If everything is a list or a map, we can create programs that write programs. In other words, the tools we use to compose an entire system together is the same tool that we use when we are building simple functions.
So what’s the answer?
I don’t have the answer. These are just some things I’ve been thinking about as I dwell on what we take for granted and what assumptions about writing code we should be challenging. There are some ideas though I’d love to play around with.
Emergent Behavior via small agents. I’m very intrigued reading the book Society of Mind. The idea that larger behavior is actually emergent and is broken down into smaller agents that work together for a desired uber outcome. I like thinking about what this idea applied to an everyday programming language might look like (ala a combination of green threading/mini-processes, messaging-passing, etc.).
Semantic data vs. Syntax. I’ve already beaten this one to death in this post. But I am curious what a programming language would look like where defining data structures was more akin to RDFs + Tuples. Instead of accessing a property via a path that is dependent on the data structure itself, you could simply ask what field you want, and the semantics of that field would be used to find the value in any data structure passed in regardless of structure (if the field exists within it).
Linda Tuplespaces. Originally found in the coordination language Linda for building distributed systems, I like the idea of a Tuple Space and Blackboard Systems. This is not just for distributed communication either. I wonder what it would be like if the local data structure experience in a language and the composing of functions were based on a “micro Tuple Space” that captured semantics about the data instead of just the syntax. I wonder if procedure parameters as we use them today are simply an artifact of the von Neumann computing model and the way they’ve been derived from a fundamentally stack-based model.
Image-based development. The first things I remember blowing my mind about Smalltalk was the idea of the portable image that follows you around and Smalltalk being an entirely self-contained operating system within that image. You programmed your system within the image itself and when you needed to ship the application, you simply shipped a (potentially “cleaned up”) version of the image. This idea of a self-contained system can also be found in Project Oberon from Niklaus Wirth and in several other places. It used to feel like a totally foreign idea to me. But now with Docker containers and container orchestration, it’s not so foreign conceptually anymore. So I could see the time for this enhancement coming in the future. I know I’m certainly curious about a language where all development is done in a self-contained docker image and the act of shipping it is just committing the docker container and shipping it out. And not just code within the container, I mean a full-blown environment like you find with Smalltalk, Oberon, etc.
Anyways, just a rant that I’ve been thinking about lately as I question how we are building systems today and what we take for granted.