LLMs, 3D printers and amplifiers

"Time to add a feature"

Imagine that a senior software engineer named Dave is working on an application that enables his team to manage configurations for various services.

He made an analysis "buy vs build" and it turns out that building a custom solution is the best option for his company, as 20% of capabilities are quite unusual, whereas 80% are standard features that can be found in off-the-shelf products.

Dave just wants to satisfy the needs of his team and deliver value as soon as possible.

The goal is not to build a generic, reusable product, that can be used across the entire company, but to deliver a solution that works for his team, so far.

He has a decade of experience where he built various systems, he worked with various technologies and paradigms.

He knows about the designing, various patterns and different practices.

He specifies the first feature, then asks an agent to satisfy the requirement using the first test.

An agent runs the tests, sees that they are failing, and produces the easiest possible code to make the tests pass, as Dave instructed it to do so.

The test becomes green.

Dave stops and looks at the code that the agent produced.

"Should I simplify it now or proceed with specifications?", he wonders.

He decides to proceed with the next specification, and simplify later.

He increments the feature by using services of the agent, following the 3S workflow.

When all tests are green, he decides to review the code and simplify it.

The agent didn't get abstractions right, so Dave specifies an internal quality requirement to improve the design by restructuring the code, by introducing proper abstract ideas like ConfigurationHistory, etc.

The agent does refactorings, and the code looks much better now.

Dave tells to verify if specs are satisfied, the agent does that and all are green.

He also sees that in some places a factory design pattern might be beneficial, so he specifies that as well.

He tells the agent to satisfy the new specification, and the agent introduces the factory pattern in the code.

Dave asks the agent to run verification again, and all tests are green.

He is happy with the result.

Just to ensure the quality of the specifications as tests, Dave decides to run mutation testing cycle to see if the tests are strong enough.

The mutation testing shows that two tests are pretty weak against mutants, so he specifies new internal quality requirement to improve the tests.

The agent satisfies the new specification by improving the tests, and now the mutation testing shows that all tests are strong enough.

Dave is happy.

He went to another feature, following the same 3S workflow, by using the services of the agent.

All took less than a couple of hours to have a walking skeleton so that his team can start using the application and provide feedback.

"We need to analyze the log files"

Imagine that a senior software engineer named Dayanna got asked to help support team to analyze log files from various services to identify issues and patterns.

She opened the 10GB log file, and started to analyze it using her favorite text editor.

She quickly realized that it is going to take ages to analyze the file manually.

She remembered that she recently read about "AI" that can help to analyze text files, so she decided to give it a try.

She wrote a prompt that instructs the "AI" to analyze the log file, identify patterns, and summarize the findings.

It produced a summary, but it was not very accurate.

Actually, it missed very important patterns happening in the domain of the application.

Dayanna realized that she needs to provide more context to the "AI", so she decided to feed it with domain knowledge.

It got a little better, but still not accurate enough.

She decided to build a small tool that can preprocess the log file, extract relevant information, and provide it to the "AI" in a more structured way.

She knew how to work with .NET and C#, so she quickly built a console application that reads the log file, extracts relevant information, and formats it in a way that the "AI" can understand better.

Still it was pretty textual for her to get a glimpse of the patterns so she decided to visualize the data.

Dayanna has heard about HTMX and decided to use it to build a simple web application that can visualize the log data.

She specified the requirements for the web application, and used the "AI" to generate the code for hypermedia-driven UI using HTMX.

She integrated the preprocessing tool with the web application, so that she can upload the log file, preprocess it, and visualize the data in a more interactive way.

She was able to identify patterns and issues in the log file much more easily now.

Dayanna decided that she could deploy this tool to help the support team analyze log files more efficiently in the future.

She knew that the application was produced in a prototype spirit, so she definitely knew that internal quality might not be best possible, but it was good enough for the purpose.

If she gets a positive feedback from the support team, she might consider improving the design and architecture of the application as the next step.

Also, she decided that it might be useful to employ a simple machine learning model to help with the log analysis by suggesting important features/patterns in the given logs.

So Dayanna specified that problem to the "AI", got four recommendations, and decided to implement a simple solution using one of the suggested models - pytorch one.

She didn't get the analysis done, but she easily produced a tool that can help with enhancing the log analysis capability of the team.

We are so cooked or we are cooking?

Metaphors.

Who does not love a good metaphor?

We need metaphors to understand, to communicate with other beings so that we are capable of helping each other, work toward a common goal (and more, of course).

A metaphor is like a bridge that connects two different ends, enabling to convey the caravans full of ideas from one side to another.

That was a metaphor about metaphors.

As you probably know, dear Reader, I love metaphors.

Some might say they are dangerous, as they can lead to misunderstandings if the metaphor is not well chosen or if the audience interprets it differently than intended.

But can we really be human without producing and consuming metaphors?

"Ok, but what does it have to do with those two tales about Dave and Dayanna?", one might ask.

I am so happy that you asked that question! (or at least, I believe you did, dear Reader).

Recently, we constantly hear that AGI is "just around the corner", and that "we're cooked", and so on.

Instead of throwing more fuel to the fire (next metaphor, huh?) of such discussions, why don't we think what "AI" can give us back, when used properly?

"Used properly" is the key phrase here.

Amplifiers, not replacements

So far, we've followed two tales of imaginary senior software engineers, who used "AI" in different ways to achieve their goals.

What is really important is that both of the imaginary engineers were experienced, reflective, conscious professionals, who knew what they wanted to achieve.

They probably had a good mentals models, sourced from past experiences, saw both "bad" code and "good" code, and were able to judge the quality of the produced artifacts.

By observing Dave in action, one might say that he already internalized TDD workflow (which I like to call 3S) and he used it pretty confidently in the past.

He used "AI" as a tool to amplify his capabilities, to deliver value sooner, knowing that the tool he produces might be actually put into production quite quickly.

It might be that Dave already worked with Outside-In development (as we saw how we focused on walking skeleton first), Domain-Driven Design (as we saw how he spotted missing domain concepts like ConfigurationHistory), and various design patterns (as we saw how he introduced Factory pattern).

For Dave, "AI" worked as an amplifier of his capabilities, to deliver value sooner, while keeping the quality of the produced artifacts at a satisfactory level.

As sound amplifier ehances the sound signal, it does not replace the instrument producing the sound, but it makes it louder, more powerful, so that more people can hear it.

3D printers, not manufacturing plants

By observing Dayanna in action, one might say that she was able to identify a problem that needed to be solved.

She didn't just jump into the solution by going through the logs, but she first analyzed the problem, identified the pain points, and then decided to build a tool that can help her achieve her goal.

She used "AI" as a tool to amplify her capabilities, to build a prototype quickly, maybe a little bit dirty, not really knowing if it will be useful later or not.

Years ago, doing this kind of prototyping and building a specialized tool would take days, if not weeks, sometimes involving a small team.

Requirements analysis, refinements, and so on.

But this time, it took couple of hours to have a working prototype that can be used to analyze the log files.

Dayanna consciously stated it is a prototype which might have internal structure that might not last long.

To judge the quality of the produced code, she probably had a good mental model of what "good enough" means in this context.

She probably could tell what pain points the prototype might have, but she was ok with them, as long as the tool helps her achieve her primary goal - to help the support team analyze log files more efficiently.

Dayanna used "AI" almost as a 3D printer - to quickly produce a tool that can help her and others, knowing that it might not be used across the whole organization for years.

Personal 3D printers are not manufacturing plants - meaning to achieve the economies of scale, producing large quantities, providing high quality outcomes.

Repeatedly.

Tools amplify capabilities

"The form follows the function", as the golden saying goes.

The usage determines the mental form "a thing" takes.

People are using LLMs for a variety of purposes, from generating summaries up to producing videos of cats singing songs.

Haven't we (human beings) always engaged with models?

Probably because of the LLM current context, one might conclude that by saying "models" I mean exactly LLMs - which is not true.

I am thinking about all the mental models we wield in our models repository (wherever it is, does not matter for now) so that we can use (consciously or unconsciously).

Internalized mental models frame how we think of the tools, hence impacting how we use them, including LLMs.

Imagine a scenario: whenever a problem appears (let's assume it's the right problem) and we built some initial understanding of its essence, we could ask ourselves, before jumping to finding a solution: "what options do we have?".

"what options do we have?" - this innocently looking question implies the mental model itself - "the operator" needs to know about the concept of "options" so that it can be used (consciously or unconsciously) for working with the challenge in front of them.

And models are like metaphors - they help us to understand, to communicate, to work together.

Going back to our tales, both of the engineers were capable of either following TDD workflow or building a prototype tool, because they had the mental models in place to do so.

They already internalized those models, so they could use them consciously to achieve their goals.

LLMs enhanced those capabilities, amplified them.

One can use LLMs to amplify skills, or as 3D printer to produce tools to amplify other capabilities.

Humans are doing that for ages.

If the way we think (meaning, what mental models use have) shapes the frame of how we use tools, then wouldn't it make sense to work on (better) mental models?

Turns out that Inverse Bruce Lee maneuver might be one strategy (a mental model?) to follow that path.

There might be more.

I invite you dear Reader, to reflect on your own mental models, observe the patterns of your thinking, and see how they impact your usage of LLMs (or life in general).

If something is not working in your measures, maybe it's time to revisit your mental models, and see if they are serving you well.

This means we need to learn how to discard or deactivate models, remembering about them, but not blindly following them, because "everything is a nail, right?" instrumentation problem.

But the real foundation is to be aware - nothing really happens if you don't see it.

"The problem is that I don't see the problem", as the saying goes.

Gigantic repository of mental models would be quite useless if one is not aware of the thinking patterns, behaviors and mental habits, isn't it?

Then, work on your awareness, dear Reader.

There are plenty of easy tools like one-minute mindful sitting on the chair, just observing your breath, or mindful body scanning to put all attention on a little finger of your left foot (funny thing, you just realized that you have a left foot, didn't you?).

The observation (the process) is the foundation of everything.

Happy exploring and modeling, dear Reader!