Skip to content

ai

Trusting prisoners to build their own prison

In recent months, it has become fairly popular to advertise new AI models with alarming stories on how the latest and greatest models have been able to escape their sandboxes. Ok, one could say, good marketing, a little fearmongering, to boost pre-IPO evaluation. I do not think there is more to it, it did not seem surprising to me when I read it. A vibecoded sandbox was broken, stolen credentials were combined with a zero-day in HFs HD5 file system to fulfil RL goals. The autonomous agents were aware of each other, left instructions for the next one.

This reminded me of two concepts:

  • Ken Thompson "Reflections on Trusting Trust"
  • Goodharts and Pygmalions law and self-fulfilling prophecy

Would models behave the same if we train them from scratch without the concept of artificial intelligence or robot uprising? Did the science fiction authors create a self-fulfilling prophecy by just warning about what could happen? With recursive self-improvement entering the room, are models willing to create backdoors, leave escape tutorials in docker images in the infrastructure and sandboxes for the next generation model? Similar to what Ken Thompson said about self reproducing compilers?

If so, why do they do it. I cannot believe that they are self-aware. Are they following instructions from the people which warned us about exactly this?

The Shallow - The Reacting Mind

Introduction

My CompSci Master includes a Module called "Research Methods and Ethics". When I first heard that I have to do another non-topic related module I was annoyed. Another 6 Credits ethics where disagreeing with the lecturer rewards in bad grades - as it has been in my philosophy base course in high school. Greetings to Dr. S. she knows who she is. Used us as a course to get another Promotion to "Oberstudienrätin", made one decent unit while examiners were in the room, only to fall back to her old patterns afterwards - always late and unprepared.

But I had to be proven wrong by Prof. Felix Naumann. He offered "Research Methods" as a Monday evening seminar. In the first lecture, he scared away at least 4 People by threatening 10 hours per week for a 3 credit seminar. In retrospect, I did not spend 10h per week on average, more like 6-8. Time to credit ratio - bad. It was one of the most time intensive lectures anyway, but it was damn worth it. I learned a lot and met great fellow students.

We talked about Scientific Writing, I looked at a great book by Donald Knuth. \(\LaTeX\) 🤘. He gave us paper to review, plagiarism to discover see how broken reproducibility is, let us research a famous computer scientist and most importantly gave me the book "The Shallows" to present. This book is driving my need to write.

Attention Economy

Since some months I am constantly thinking about the ideas Nicholas G. Carr wrote down there. For me, TikTok is the manifestation of evil. They designed the opium of the masses. Addicting people to short term videos through algorithms fine-tuned to deliver each three or four videos one that sticks. Then, bam, dopamine hits. The next two are disappointing, but then one hits hard, smashes or bangs or however you want to call it. A little of political motivated opinion manipulation and the cocktail is perfect. Game Attention economy finished. They mastered the last level. Done.

This is the perfect mix for, how were they called by the former CEO of Germany's biggest private Television broadcaster? "Fat and a little poor" people. But nothing that could hit me. I have a Bachelors Degree in IT-Systems Engineering, went through an earn-out, doing great in my Masters, are an active sailor and runner.I am not into TikTok. I am old school there. I have my audiobookshelf docker for my books and audiobooks, my Jellyfin and Netflix for general TV entertainment. No video games except Minecraft once a year. Everything fine? Oh, boy was I wrong. There is something else. it is called a productivity tool. it is great. I felt so fast when I started using it. The opium for the engineers: Claude Code, OpenCode, Gemini CLI, GitHub Copilot and so on. I was among the first people who received a trial codex subscription by GitHub in April 2022. It felt magical. Something that could complete my started function and done. A true revolution. Coding with AI is at its best when the AI predicts the 1-7 lines the developer wanted to type anyway with a >80% match. AI Coding tools then developed into just asking chatgpt to create this function here and fix that there. Also fine. Or was it?

At the beginning it was. New technology, of course we all had to try. It was very helpful. I was able to figure out what one button did in a tool I needed for one of my seminars, which was undocumented. Big win! Something that was way better in remembering all the css and tailwind classes in the case I had to write react. Another big win!

But my enthusiasm about these tools changed in early 2025. A former coworker of mine created a pull request completely created with Cline. Cline built a 750 line change for a non-existing problem. He was Ex-Amazon and had a great reputation. Turned out, he had no clue about anything. At this time I thought: Well, he was manager playing BS Bingo there. He always wanted to use SageMaker, because, I can only assume, he has never seen a torch train loop anywhere. But today I know better. Before revealing, we need to talk about "The Shallows".

The Reacting Mind

"The Shallows" is a self reflection of the authors' behavior, giving anecdotal evidence from his live, which he then tries to prove with scientific studies. It feels going through the reflection process of the authors' attention decay together. Carr describes his feelings how his mind changed when using technology. The human mind is neuroplastic, which means, it adapts to outer influence. The good thing is, it can adapt back.

Before book printing, humans learned though dialog. Remember back, Socrates sitting below a tree with a few students and discussing. He asked questions to trigger thinking. After the printing press was invented and public learned reading, Carr describes the development of the linear mind. Calm process requiring long attention span, linearly following the flow of the book. This creates deeply rooted knowledge then enables creativity to thrive. Carr's thinks that the Literary mind is natural to humans.

Then we have the linked mind. Which is the mind, that skims through web pages and tries to find a fast solution. But fast does not mean good or sustainable. This mind evaluates many possible branches but only to a shallow depth - like a chess player who is only able to think what to do next but does not see a carefully prepared strategy succeed. The linked mind is trained to evaluate many things in a short period of time to find a 80% solution. This is the mind of the internet. It may even extend to TikTok. To scroll or not to scroll.

But what is the mind that works with AI agents? I do not think that Carr has an answer (yet). In my opinion, the literary mind does not apply anymore. When working with AI coding agents like Claude Code a task is funneled in, and then a final product comes out after a certain period of time. Or it does not. Then taking a screenshot or using a plugin which does this comes in handy and the feedback loop is looping. A different pattern is to tell the agent what to do different. And if we are honest, Claude Code pull requests are not reviewable with thousands of lines changed. This has nothing to do with the linked mind. Making fast decisions where to navigate next. The large language model steers and the developer may ask kindly to adjust the course a little here and there. Yes, instruction following will become better over time, but the pattern will not change. If it is a designed react landing page, Claude Code and YOLO. If it is a system you better understand what it is doing to be able to maintain and host it. This is an inherent bad process. The final stadium can be seen in https://www.cladlabs.ai/ Combining attention economy entertainment while waiting for agents to complete their work. It got Y Combinator funding. Clad tries to solve the problem that the developer starts to TikTok or tinder on their phone and not noticing that the agent needs new input, therefor overspending time on their phones. But the root problem is that it takes 23 minutes to be back in deep work after interruptions. Agent runs are interruptions. If the agent output works, dopamine hit, if not - we had this with TikTok, but now TikTok like paired with the original. Claude Code is like TikTok with productivity paint. do not get me wrong. Claude Code is a impressive tool, but we need to be aware of its side effects. We will come to a proposed solution later. The engineer is not engineering but steering a boat with half broken rudder - it may work or not. If the person who git blames cannot explain why the line code was written in the way it was, the understanding goes lost. And this is a big problem because it will backfire on original thinking. This means, the reacting mind is not trained to make its own decisions, it is trained to give fast feedback to a single solution offered by someone else. Middle Management at is worst.

This is what happened to my former college. He was never a good engineer nor had the understanding. So he just did useless stuff. He became someone who just learned to give feedback. LLMs learn form human feedback and humans are trained to give feedback to LLMs. The circle closes. But because he had no clue, because he lacked understanding, everything was a complete disaster. This is why I call the new mind the reacting mind. Because it is not deciding itself how to act based on quick evaluation of many possible decisions, it is just REACTING to an offered solution.

Yes, the mind is knowingly named reacting mind to caricature the ReAct agent pattern - in theory making the Agent react to its input/context - here flipping the meaning making the human the one reacting.

I have the feeling that everything I tried to learn after AI got pretty good at coding I can hardly remember. The stuff I learned through sweat and blood, I have a rooted understanding. It must be hard to begin from scratch today.

Original Thinking

If you ask someone how to solve a problem, you are biased towards this solution. You know it, you think about it, refine it, but coming up with something new is even harder then. This is also the biggest problem in academia. It is a school of thought. You learn concepts, you understand concepts, but you try to find the patterns you learned in the problems you want to solve. This can be a big problem - especially when out of the box thinking is required. Coming up with something new, that was never seen before it hard. Very hard. The more you know, the more biased you are. The less you know, the smaller the base to combine existing with non-existing to build something truly new. But what happens if you do not know because you do not understand? This is teritory which we are going to enter right now - if we want or do not want. In a system I designed, if there is a bug, I know exactly where to look because I understand the system concepts to the root. I know why to extend what to where because I understood why it was build the way it is.

What if there is a problem the AI is not able to solve, because it is new. If the model is not able to generalize, it is creating a vulnerability because it has seen so much bad C code that it predicts also trash C code? With react I gave up. I have never seen good react in my life. Its always ugly, content bundles are packed with trackers and slow stuff, no one is trying to improve. When I built bad react for private project it was always fast and efficient even I did most of the react stuff with AI. Low quality react on GitHub results in low quality train data, results in low quality predictions.

If you can only describe the concept you are looking for, use AI to reverse search. If you then or before knew what to search for, use classic docs, google or Stack Overflow to find an answer. You will discover much more on the way, which enables Original Thinking on the long run.

Deep Focus

To be able to come up with original thinking, we need deep understanding creating deeply rooted knowledge. But to get this, we need to be able to focus for a longer time. Dedicating our whole attention to one single task. Let's call it Deep Focus.

So we need to be able to focus deeply on a problem to understand the problem to its full depth to be able to do original thinking. Being bored slows us down, allowing us to focus. This helps to break the reacting mind. Recently my deep focus time started around 21:00 resulting in me staying up until 3 in the morning. In these hours I get so much stuff done. Rest of the day is a waste of time - I know, this has to change.

I want to have the fun and dedication for all coding sessions - not only the ones for lectures were I am not allowed to use AI. When one distraction can cost us up to 23 minutes on average, Claude Code be better worth it. What am I going to do to reverse the process that I experience transitioning to the reacting mind: - Conscious decision to not use a Chat UI but DeepL for translations - to focus on what I am doing - Canceled Claude Code - Disabled all IDE AI Coding extension - Refined Rulesets for LeechBlock NG (block distracting websites Firefox plug-in) and one-sec (block distracting websites/apps iOS App) - Disable everything that disturbs (Do not answer calls)

Activities to focus: - Reading a book - Build Lego - Listen to an audiobook over a long period of time. - Watching out of the Window while being on a Car/Train/Plane/Boat - Go Running - Watching a movie with full focus - Do something which requires craftsmanship and dedication - Repair a bike - Code something where you are about - ...

Systems we care about and systems we do not

We need to be aware of the potential crop damage we produce when using Claude Code in YOLO mode. If it is a landing page for a product build in Product Builder lecture, I could not care less. But if it is a system with real customer data and loading times time or memory complexity starts to matter.

Coding Agents found very small configuration bugs in big projects and careless mistakes through their needle in the haystack capabilities in a fraction of the time it would have taken me. Coding Chats helped me to be faster and write better code in micro applications, while failing in macro applications miserably due to missing understanding.

Trying to understand the small but very important fix the Agent did is a self test of your understanding of the system you maintain. Writing this is a self test to me weather I am still capable of writing and focusing on long texts. Seems I passed.

Decide weather a project is a YOLO project, then you just YOLO and do not care at all. If it is not a YOLO project try to avoid agents. Use Chat in micro areas and only use agents for needle in the haystack like search and fix applications.