AI is everywhere so I thought I’d see what I could do with it productively.
Working with LLMs has been an interesting experience. My team:
Me: Architect and Product Owner
Claude: Lead Engineer (Claude Pro account)
Codex: Senior Engineer (Codex Pro trial)
Copilot: Intern (GitHub free account)
Along with setting the project guidelines and directives, managing token spend and task allocation has had its challenges. Keep in mind, I am working with a limited budget.
More than that, being the human in the loop has been quite taxing. LLMs produce code prolifically. Unless I keep the tasks chunked small enough, the AI still strays from the directives:
APIs/contracts/abstractions first, implementations behind them
Tests at the API level, spec first
Run tests locally, then staging; nothing reaches the prod registry until it’s scanned and signed, including public libraries and containers
Jira tickets are the source of truth for what is designed and implemented, and Confluence pages are composed from Jira tickets for the publishable design. This isn’t standard process, but it’s one I’m championing. It would normally be preceded by a concept doc, but that one lived in my head.
Use variables and string replacement: be ready for i18n
…and more
Despite these guidelines, LLMs don’t always stick to them: working context, compaction… Some LLMs go down the rabbit hole more often and burn through their tokens; some are very efficient. Sometimes they do something in one session and fail to do the same in the next. Sometimes they forget. Sometimes they assume. Lots of challenges like that.
In the end, though, LLMs are scary good at writing code. They write a lot of it, and they can do a lot with little guidance.
I handed the AI the autonomy and kept the accountability with me.
Without an effective human in the loop, the code can spiderweb and bloat (I’m sure this will improve; the rate of change is astounding). And AI does make mistakes. I can’t count the number of times it has said “that was my mistake” or something along those lines. But it does recognize and acknowledge them, and the ratio of errors to productive code is small. Sometimes your lead engineer launches into a complicated task and you just have to wait and hope you can assess what was done at the end…
This has been a roughly 6-month project, with a lot of ramp-up to establish processes and guidelines. Most of them had to be discovered along the way and then embedded into the project directive. Only recently did I add more AI engineers and expand the development process (staging, prod environments, etc.).
Leave a Reply