Token costs and context management are becoming more and more of a problem. The stories about Uber spending it's whole budget in four months and AI labs (finally) finding product-market fit Funnily enough, we might already have a fairly good model for solving at least part of this: human organisations.

Tokens might be mapped to effort and context to, well, information access and wider context. There is a balance on limiting context so that the tokens / effort produced isn't out of sync with the value of tokens produced. A complicated way to say that adding more context isn't always worth the costs. The way organisations deal with this is breaking jobs into smaller chunks, and limiting the knowledge to what is needed for that job[^1]. And in some way the capability to work with context is also tied to how complex the job is.

The same should work with agents. More capable and expensive models should be used for those tasks that need that additional context (i.e. tasks which are ambiguous or need broad context). Smaller and cheaper to problems with narrow scope. That limitation is actually part of the benefit. A narrow agent does not need to know everything about everything, rather it needs to know everything about this one thing.

This is not exactly a new insight. We have had decades, if not centuries, of experience in building organisations around delegation, specialisation and coordination. Most real organisations are obviously nowhere near the ideal version of this, so we shouldn't just implement any organisation model as is. But designing agentic workflows might very well be a return to organisation design

[1]: and yes, sometimes that's taken too far. And with humans we obviously have other concerns like right to know things, which aren't strictly tied to the performance of the task. I can do shoveling of a dig without knowing the company vision or revenue forecasts, but I have right to know those nonetheless.

Simon Willison gave Fable a task and it went wild. Seeming as the costs are on everyones top of mind, this kinda seems, well, not ideal?

New models, new era? The initial impressions from influencers like Simon Willison and Ethan Mollick seem to be highly positive. My own testing has been so light that no real insights yet, need to push it a bit more. Beyond the pure capabilities the most interesting part is way Anthropic is rolling it out: Fable will not do everything, and might not even say to you it doesn't do it and Anthropic is saying it forces 30 day data retention and will review all content. Anthropic is saying AI is dangerous, but will still develop and release it while saying that everything is ok while they have control. Meanwhile some users have had access to Mythos already for a while.

Apple AI. Lot of slightly interesting AI features (which this time seems to be really coming), most of which apparently are not coming to EU. Apple is pushing hard against EU legislation, and partly for a good cause. EU is essentially saying to contradicting things: privacy is key, and interoperability is key. The conflict being that interoperability across different providers will necessitate sharing data between those providers against privacy goals.

It’s not a surprise that Apple chooses to respect the privacy point, and not the interoperability. This has been Apple’s MO for the full Cook era and obviously serves their short-term business objectives. But the world is moving on and it remains to be seen if Apple’s strategy is correct.¹

Footnotes

1. Regardless of Apple’s strategy, the EU legislation would need clarification and something to solve the tenaion between privacy and interoperability. Not the easiest task, but necessary. ↩

https://www.youtube.com/watch?v=Uvl-tRga98g

Peek into how Anthropic Labs works. No documents, only prototypes when building with Claude and small teams doing holistic work without strict role boundaries.

I wonder how much of this is possible due to the nature of products Anthropic does as well as their funding situation, and how much is about the optimal way for working with AI.

I had something of an epiphany¹ while creating this site.

Every single line of code² on this site has been written by AI. Mostly with Codex but with Claude as well when I ran out of tokens. The original idea, two years ago, was to test how simple it would be to get a static site up and running. I got it working, but I just didn't use the blog at all and left it rotting on the Internet until a few days ago when I got an itch to do something with it.

While I am somewhat of a heavy user of AI, refactoring the earlier site really made it tangible how much the AI workflow has changed in the last two years. From chat & copy-pasting to building with agents is more than a vibe change, it's the difference between being knee-deep in code blocks and letting an assistant handle the dirty parts while you try to decide and express clearly what you really want³.

Instead of chatting and doing cmd-c/cmd-v between ChatGPT and VS Code, the new way of working is about brainstorming and planning with AI in the terminal⁴, with execution largely hidden. Getting different options and changing anything from layout to publishing workflow is almost too easy. What would a different layout look like? Just command⁵ AI to create four different options and choose one, or discard all of them.

It would be easy to say no one is going to need professionally built software anymore. In fact, after all this I have more respect for professional developers who are able to direct agents to a more robust solution without hours of back-and-forth. Solving those "pesky" little problems like syncing across devices and protecting content integrity is a lot harder than it sounds.

I might feel differently in two more years, but I still mostly want to use apps someone else made. A well-designed and thoughtful app made by someone who cares about it is a joy.

Empowerment doesn't take away respect for professional work.

Footnotes

1. or just coffee overdose ↩

2. but none of the content! ↩

3. yes yes, this is a simple static site, more complex applications are still a different thing. For now at least. ↩

4. I did originally try the Codex app as well as Codex on the web, but using the terminal was just so much easier. Less secure, but easier. ↩

5. command, not ask. This is another learning: asking nicely takes much more tokens compared to simple commands. ↩

Roger Wong writing about the Uber COO saying justifying token costs is getting harder

When factories converted from steam power to electricity in the 1880s, they swapped out the engines and did nothing else. The floor plan and workflow didn’t change. For three decades, output barely moved. Only when companies redesigned their factories and process around the new technology did they see an increase in output.

While the analogy doesn't fully work (not sure whether replacing steam with electricity doubled the operating costs) but the message is clear: until we redesign the ways of working, we won't see the full benefits. I have been thinking a lot about this especially regarding async coding practices and mainly about decoupling the human time from the coding effect. Then again, Dan Shipper was just arguing for tight coupling agents and humans so maybe the redesign is something else.

After maybe a year of thinking about this, I finally decided to post at least the introduction post. Idea would be to occasionally write short thoughts & link to interesting articles, let's see how this will go.