Why AI Will Never Understand Why
On Context, Connection, and the Irreducibly Human Nature of Engineering
Introduction
At my core, I am a technologist. Good software engineers have to be technologists. We should continuously be discovering and creating, while being in awe of what others are discovering and creating. We should be constantly learning, testing theories, and trying new things.
Often, we can get into a lull and feel diluted by the day-to-day monotony. We lose sight of our mission or our company's mission - which, no matter what the actual wording is, should boil down to equipping people to be empowered through technology.
In the current technical climate (ahem, AI), it is more important than ever that we remain uniquely human in how approach technology. People are seeking connection more than ever as we've been hidden behind our screens. Maybe we are at a pivot point, maybe we're not, but we all have been trained to interact with our technologies in ways that the simulation of LLM generated responses have also been trained on.
Stepping Back
Do you remember your first interaction with programming a computer?
My introduction to programming began with QBasic and messing with the Gorillas.bas game. But my first real programming experience, besides some Geocities websites, was building AOL "hacking" tools. These were tools that would help you spam chat rooms, send mass emails, and crash other users AOL program. Somewhere in my parents' house I have a backup of these, but probably most are lost to time.
These were Visual Basic 3 programs, written in a stolen version of Visual Basic, often based on other developers decompiled code. There were methods of reasonably decompiling Visual Basic applications to learn how they worked, modify images and buttons and functionality, add custom sound effects and messages, and distribute them for others to use.
One day, a new single message "punter" string was discovered. I remember how pissed me and my friends were that we hadn't thought of it first. But, you try new things, learn from others, and mess with people on AOL. That was what it was all about.
All that leads to where I am today. A couple years after his, I would pick up some C/C++ and do some Y2K work in COBOL. I didn't start writing code from scratch, I built on top of the work of others. It's like that [XKCD comic](https://xkcd.com/2347/) where some piece of functionality that some guy in Nebraska maintains is in essential building block for large systems. We learn the same way. We search StackOverflow for solutions, we see a cool repo and fork it, we use frameworks and abstractions and intellisense and boilerplates.
Why is that?
At the root of this is that we are trying to solve a problem that is unique for us. Because, here's the thing, and I think this is true for every software developer, whatever company you work for, whatever the problem is, there always comes a point where it feels like we are the first people to have to solve a specific problem in a way that no one else has before. Every software engineer eventually faces this same realization - we might be the first to attempt to solve this specific problem in this specific way. This feeling is at the heart of what makes engineering irreducibly human.
Why is that?
The question "why" is the reason. Why are we trying to build something the way we are? Why are we trying to address our users the way we are? Why are we trying to automate certain things? Why are we using these coding standards instead of others? Why are we using this framework versus another?
The "why" is what makes our problems unique - because we are unique individuals working in a particular environment at a specific time with other unique individuals in the same situation. We all have strong opinions, we all have a way that we have done things in the past, we all are looking at a problem with our own perspective.
As an aside, this is true outside of engineering also - within the departments at the companies we work at and just life in general.
Here is an example:
A company needs to start tracking their customers. Currently, there is a workflow that exists in various spreadsheets and SOP documents, but it is a mostly manual process. The company decides to build a custom CRM. Now, there are plenty of CRMs out there - the company could use Salesforce or Zendesk. However, no matter which third-party software is selected, the team would have to adapt their current workflows to matching the workflows from that software. The problem is that the current workflows have been created and refined to meet specific company goals and reporting. Adapting to a different workflow could jeopardize those targets.
So, the company decides to build from scratch. While maybe 80-90% of a third-party program works for the existing workflow, there is that 10-20% that is unique for the company. This all factors into the requirements for the project, which then makes its way to the software engineers, and they get to the same situation, they are the first to try to solve a problem in a certain way.
Again, this is like life in general, right? The systems in place around us would work perfectly if everyone fit certain criteria, but we are all different, coming from different places, working toward different goals. And so we adapt and grow and modify and start from scratch. The way I interact with my coworker is different than the way I interact with my neighbor. This is because of the different contexts that we are in, but also because my coworker and my neighbor are different people.
Now, AI
And here is the overall point, and brings us back to the "why" - AI will never understand the underlying "why" of a goal or situation. It will simulate an articulation of the why, but it won't understand the why. LLMs and Machine Learning models produce approximations of data in order to estimate a response in words.
The current hype around AI replacing software engineers misses something fundamental about what we actually do. Yes, we write code and much of that code follows patterns. But the core of engineering isn't in the specific syntax or pattern, it is in understanding the why behind what we are building.
This is because LLMs will always try to give an answer, knowing or not knowing isn't a part of its approximation of an answer process, even in reasoning models that may attempt to improve and validate the pattern matching and approximations.
The reason we articulate the question of "why" is because of and so that we can "know" in the ontological sense. The AI models don't "know" anything the way we as people (especially technical people) know anything. Anything we know is based on more than facts. We "know" based off of experience and context and subtle micro expressions and a whole host of other interactions.
We build software based on what we know, addressing the why that comes from our goals. We can provide some context to a model, we can provide some goals, but the machines will never understand the context or goals.
And maybe I will eat my words here in a year or two, we'll see. Some may say that this way of human "knowing" (or understanding) is also just pattern matching. Maybe if we had enough compute capacity and memory, an AI system would be able to pattern match as good (if not better) than people. However, if knowing was simply pattern matching, knowing would just be storage and repetition of facts. Real knowing (or understanding), though, comes from facts that are discovered as a result of experience. This may be the application of previous experience to studied facts or the trusting of others' experience to prove facts. This level of experience can only be actually attained through the lived life experience of a person. There are no approximations of this experience that wouldn't take the amount of time that a person has lived and experienced the world around them.
As software engineers and technologists, we know that there are all sorts of scary stories about LLMs taking our jobs and all the wild things that it can do. But at the end of the day, it is a tool like any other tool. A tool in the hands of the right professional makes all the difference in the quality of the outcomes that it produces.
AI is not going to take our jobs. But it will change our jobs, similar to intellisense and autocomplete and build tools and new IDEs and whatever other tool has come along that we have adapted to. These AI tools are based off all the technological knowledge worker progress that have built our computer programming experience over the past 75 years. The principles of software engineering and computer science, however, remain the same. We build systems and solve problems and utilize the tools that we have available for us to meet those goals.
This is specific for our roles as software developers, but it is true for all knowledge workers. One of the key differences with AI is its apparent ubiquity and application across all knowledge based work. We, as software engineers, are, and should consider ourselves, prime knowledge workers. The technologists are the ones who will understand the shortcomings and the application of AI tooling best.
A related example might be, similar to above, how an operations team's process requires some skillset in spreadsheets. Where once this was an optional skill, now knowledge workers in operations are required to understand spreadsheets. Which, 30 years ago was a hurdle, now we all grow up using spreadsheets to some capacity and don't necessarily need to be trained in how to use spreadsheets.
Over the past 25 years, we have seen most all companies transition to becoming technology companies, then data companies, and now we are in the midst of the next transition. AI will affect how we all work.
End
We have an opportunity, as prime technologists, to build great things for great reasons - and build them well. Our goal needs to remain the same, to build technology to equip and empower people. People is the goal and the reason we can come back to the "why".
Organizations will continue to come up against problems that require unique solutions for the same reasons they always have. These problems may have a solution that an LLM can produce, but it will produce that solution based on the unique users who are utilizing the tools. And even if it brings a solution, it will never understand the "why" the way that the individuals in an organization will.
This is because the "why" comes out of the interactions between individuals. The meetings, the code reviews, the demos, that requirements gathering, the user research, the industry experts, the personal experience, the happy hours, the presentations are all interactions between people which creates an experience for individuals who come together in an organization. These interactions are where the why is made known and where knowing has application.
Again, this is true of technical organizations and technology itself, but it is also the reality of the way our systems form and what defines how the world around us works. If we acknowledge the differences of experience of those around us who we interact with and apply that knowledge to understand the why of the individuals around us, the world works better.
This need for human judgment in complex systems extends beyond engineering. Consider the recent case of an AI candidate for mayor in Cheyenne, Wyoming. Arvind Narayanan, professor of computer science at Princeton University and author of AI Snake Oil, shares a similar sentiment about the need for human interaction. His point is that government systems work because of the process of people interacting, because of the why that exists in the conversation between two entities.
"To try to automate that is to miss the point entirely. The guy running it claims that the bot has an IQ of 155. But today's AI isn't smarter than people in any meaningful sense, IQ is not a valid measure for AI, and most importantly, whether AI is smart or not is irrelevant. Governing is hard not because of a lack of smarts but because adjudicating between incommensurable values is hard and because allocating scarce resources among stakeholders is hard."
No matter where we as software engineers find ourselves in our careers, this is the part that AI can't take over - the necessity of human interaction in the systems that we build.
And if, and that is a big if, the times comes where that connection is lost, where we give in the mediocrity of replacing people with machines, then we might as well just pack it all up. What is the point of any work that is not based on enabling human experience? What is the point of our work if it isn't built on the experience we have with others for the benefit of the people who are using our products?
Our roles and jobs as software engineers might change and adjust, but its just like any other technical problem that we are trying to address. We are uniquely qualified and equipped to utilize these tools in a uniquely human way. We should continue to deliberately build and push towards the goals, while learning and trying new things and creating experiences that are for people, applying what we know so that what we build can address the why of it all. Machines may mirror the syntax, maybe very well, but mirroring isn't knowing and misses our purpose. In the gap between simulation and understanding, patterns and meaning, is the core - the human heart - of what we do as engineers.