Skip to main content
Torta StudiosGet the guide

Free field guide

So you built a Slack bot. Now prove it can do the job.

This field guide is the full story of how we turned an eccentric internal bot into a tenant-isolated AI operator working directly with customers, and built the measurement system to know whether it was worth trusting.

12 chaptersReal production system
Get the field guide
web edition + PDF download

The complete guide: the incident, the architecture, the behavioral model, the modeling decisions, what the evidence changed, and the six-part scorecard to run against your own agent.

The guide opens right here. We’ll also email it to you; one click unsubscribes.

How it starts

The bot was the support UI.

A customer’s pipeline broke before the support interface was finished. Instead of finishing the interface, we gave our Slack bot scoped, read-only credentials and asked it to investigate.

It cross-referenced Sentry, the replica database, the pipeline state, and the codebase. Then it fixed the problem end to end: the refund, the pull request, and the deploy.

~2 min
to root cause
~10 min
investigation to deployed fix
1
pull request opened & deployed
0
support interfaces required
One impressive incident proves possibility. It does not prove a product.

The problem this guide solves

Your agent can be operationally healthy and commercially useless.

We had traces, spans, tokens, tool calls, latency, exceptions, transcripts, and job records. We still could not answer a single basic product question. If your agent serves customers today, run this test against your own dashboards:

You probably know

the machine
  • How long a model call took
  • How many tokens it consumed
  • Which tools it called
  • Whether an exception occurred

You probably don’t

the product
  • Which customers are receiving value
  • Whether people accepted the result
  • Which capabilities bring people back
  • Which accounts are quietly fading

Inside the guide

Twelve chapters. One argument.
01The bot was the support UIThe full incident, minute by minute.
02One incident is not a productWhat we knew vs. what we couldn’t answer.
03What D.B. Fresh actually isAn AI chief of staff, not a chatbot.
04One bot, many isolated worldsTenant isolation, drawn so nothing crosses.
05The wrong way to measureThe event landfill, and why it always fails.
06One Turn, twelve questionsThe behavioral atom that answers everything.
07The three hardest decisionsTurn boundaries, autonomy, internal reasoning.
08The model evolvedWhy the taxonomy changed on purpose.
09What the evidence changedSupport beat the flashy capabilities.
10What is liveProduction evidence, real screenshots.
11What it still cannot proveThe honest limits of behavioral data.
12The scorecard26 questions to run against your agent.

Who this is for

Written for you if

  • You’re post-revenue with a bot, agent, or conversational workflow already serving customers
  • Your bot sometimes fails, times out, or misreads context, and you can’t explain why some users adopt it and others disappear
  • You have logs, traces, and transcripts but no product understanding
  • You’re trying to turn a promising prototype into a reliable coworker

Skip it if

  • You’re deciding whether to experiment with AI at all
  • You have no product and no users yet
  • You want prompt-engineering tips
  • You need a basic PostHog installation
If your agent can’t answer these questions, you have an agent product problem.
Get the field guideBook an Agentic Workflow Product Review