Posts on Tal Globus

Expect It to Fail

Thu, 29 Jan 2026 00:02:56 -0800

What does the worst possible intern look like? You give them a task, they run off, and return hours later with something entirely wrong. You nudge them on why it’s wrong, send them off again, and they return with something wrong in a different way. After enough rounds of correction, they finally produce what you wanted — with dramatically more time spent correcting them than you’d have spent doing it yourself.

What about the best possible intern? You give them a task, and they return hours later with exactly what you wanted.

The detail the interns don’t generally share: both followed the same process, both failed a few times. The second one just didn’t show you the failures.

Most output we encounter in the world is like this. Behind every witty text you receive is a notes app full of discarded drafts. Behind every finished movie scene are dozens of takes with mumbled lines and poor timing. We’ve come to expect polished output precisely because the iteration that produced it stays hidden.

But AI tools don’t work this way, at least not yet.

When working with AI, it’s common for the tool to misunderstand your intent or fail at the task multiple times, with you nudging it to continue or try differently until it succeeds. This is a natural workflow when working with AI, but it’s deeply unnatural when working with people, because it’s the workflow of the worst possible intern.

The result is a gap between the opinions of new and experienced AI users. New users give the tool a task, watch it fail, and conclude that AI is incapable of real work. This would be reasonable behavior when evaluating a human expert (which incidentally is how these tools are often marketed) but it misses that AI tools are not yet capable of hiding their attempts from you. Experienced users ask the AI to keep trying, describing a feeling of “nudging” or “prodding” the AI past its failures until it completes the task. The end result is the same as the movie scene; you’re just forced to endure all the bad takes along the way.

This will partially resolve with time. AI products are already growing more deliberate about obscuring failures from users. Spend a few minutes asking ChatGPT to do complex research and read through its reasoning summaries, a large portion will consist of the model struggling with paywalls, random errors, its own moderation system, and so on. Some improvement, however, requires better contextual understanding on the part of the AI. Often the intern doesn’t fail because they can’t complete the task, but because they fail to grasp the request and the context required to interpret it. I’m optimistic about AI memory systems for exactly this reason.

But a subtler dynamic is also at play. If you expect the AI to make mistakes, you’ll want to observe what it does, and you’ll be disappointed by the failures that even a successful outcome required. If you trust the AI enough not to watch, you won’t witness those failures.¹ This creates a cold start problem: early, skeptical users see AI as far less reliable than users willing to let it work unsupervised. Trust begets more trust; distrust, more evidence for distrust.

For now, perhaps the best advice for early adopters is this: judge AI not by the first things it does, but by the last.

AI tools are still fairly spiky in the shape of their capabilities, with expert-level performance in certain types of tasks and monkey-like performance in others. To learn which is which, it’s often helpful to spend your first few weeks or months helicopter parenting your AI to develop an understanding of when you can walk away. As this isn’t good for building trust, becoming AI-proficient requires some combination of tolerance for failure and an unusally strong long-game mentality ↩︎

Doing Something That’s Never Been Done Before

Fri, 14 Nov 2025 02:03:22 -0800

I’ve found recently that I’ve been a bit hesitant to take on new projects, especially of the interesting variety, largely out of a fear that I’m doing something unoriginal. As anyone who has spent any substantial amount of time in the markets will tell you, being second to an idea isn’t necessarily terrible, but being 50th generally is, and I’d really rather avoid being 50th whenever possible. How do you guarantee, or at least maximize the odds, that whatever you’re doing, you’re the first?

It seems there are actually a few good rules of thumb to this.

First and foremost, people have lives. They only have so much time they can give a project, and the more time a project takes up, the more likely they’ll either decide not to take it on, or that they’ll run out of time and not come back to it later.

The same goes for difficulty. While I wouldn’t say that people, by their nature, are lazy, I would say that evolution has favored this behavioral pattern where people tend to be less likely to do things that are more difficult, or involve a greater number of steps. As a result, the greater the degree of difficulty an endeavor requires, or the greater the number of steps, the less likely that someone has done it, or that they’ve finished after starting.

Next, obscurity. I don’t think it should come as much of a surprise that the more obscure the thing that you’re thinking of doing, the lower the likelihood that someone has stumbled on it and done it.

Finally, there seems to be this fourth factor combining obscurity and the number of steps in cases of progressive dependency, where later steps require previous steps to be completed to even be discovered. Say one in 10 people in your industry will stumble on whatever idea you happen to have first. Then maybe one in 10 people who complete the step following the first idea will discover the second idea, and then one in 10 of those who complete the second will also discover the third. It stands to reason that the more steps your project requires, the less likely it is that someone else will have stumbled on all of them.

So if you want to truly do something that no one has done before, do something obscure, do something time-consuming, do something difficult, and do something that has unknowns you’ll only resolve once you complete the first bits. The more of these your project involves, the lower the number of people who will have done it. Enough, and it’s probably fairly likely you’ll be the first.

A Follow-up to US Government Math

Mon, 07 Jul 2025 07:03:16 -0700

I wanted to give a bit more information on how I wrote that piece. I wrote it in a very different way from how I usually write, and I want to try out this new method to see how it works. I’m sharing my process for anyone interested in writing, or just curious about how I’ll probably be doing this going forward.

I started, as many of these posts do, by talking to myself and realizing within a minute or so that I had a pretty good argument and wanted to get it down somewhere. Usually I would sit down and type out my argument, but I’ve found that typing tends to lead to a very different process from speaking. I do my best discovery by speaking out loud—ideally, though not necessarily, while alone.

So I started a recording and just kept talking. I spoke for roughly half an hour, thinking through my arguments, pausing when I needed to, and restarting twice. I ended up with a recording of everything I’d said and added notes like “maybe cut that sentence” or “make sure this fact is correct” where appropriate.

Then I fed the recording to a transcription tool and copied the transcript into OpenAI’s o3-pro with the following prompt:

Take this poor-quality transcript of a first draft of a speech, complete with stutters, restarts, and repetitions, and turn it into a single, well-drafted, cohesive version of the same. Don’t change the structure, wording, or argument unless absolutely necessary, though it would help if you could point out areas to tighten or clean up. Check the numbers and make sure they’re correct, adding footnotes with the math where possible. My source for the spending data was https://www.visualcapitalist.com/breaking-down-the-u-s-governments-2024-fiscal-year/ and my source for the wealth tax impact was https://www.vox.com/policy-and-politics/2019/2/12/18211833/wealth-tax-calculator-warren-sanders and I calculated the effect of the 10% additional income tax on the top 1% as 3.3 million (number of 1% in US by overall population, assuming that’s the denominator in the 800k cutoff) * $800,000 (the approximate income cutoff to be a one percenter, though I asked AI instead of having a solid source for that). Check my facts, and make sure I didn’t say anything wrong. Critique my arguments, and suggest improvements or additional points where necessary. Finally, feel free to add anything else to the markup you think would be helpful.

Here’s the transcript I’ve got:

(Pasted transcript here)

The draft I got back laid out my argument much closer to how I wanted it but still needed some improvements for transparency. I revised it—mostly addressing the points o3-pro flagged—then published the final version here.

The whole process took about 45 minutes for recording and transcription, five minutes for the AI pass, and another one to two hours of editing, some with AI, some without (the entire next paragraph, for example, I added manually as part of my final edit). I’m happy with where the piece ended up: it sticks to my original message far more closely than my old manual method, and o3-pro really is a fantastic writing partner.

The biggest issue I spotted is o3-pro really likes to paraphrase. I’d guess this is by design, as you don’t want to use it to write or edit a piece, only to find it plagiarized an existing source verbatim. I switched to o3 after a while, particularly after the original source analysis and error correction was done. O3 is better at not paraphrasing, but still often cut out substantive sections, or suggested edits as a series of bullet points specifying changes rather than rewriting the entire piece with edits each time around. I’d imagine both of these flaws is due to a desire on OpenAI’s part to minimize excessive token use in their fixed-cost ChatGPT product. It’s fairly straightforward to train a large language model to be more concise, and it’s certainly more cost-effective.

I’ve read the piece end to end a few times and can’t spot a section that doesn’t sound like me. I’ll probably stick with this approach going forward, partly because I’m much more fluent aloud than at the keyboard. If you’re curious: yes, this follow-up was produced in the same way, though with a lot less editing. Below is the completely unedited transcript from this very piece for reference.

Unedited transcript (verbatim)

A follow up to US government math. So I wanted to give a bit more info on how I wrote that piece. I wrote it in a very different way from how I usually write. I want to try out this new method see how it works. So I share my process for anyone else who’s interested in writing or just curious about how I’m probably going to be doing this going forward, I started as many of these speeches do, by talking to myself and realizing within a minute or so that I had a pretty good argument and I wanted to get it down somewhere. Now this is usually where I would sit down and actually try to type out my argument, but I’ve always found that typing tends to lead a very different lead to a very different process in speaking. And I do my best discovery by speaking out loud, ideally, but not necessarily, while alone, actually, maybe cut that section. Yeah, maybe cut that section. So at this point, I started a recording, and just kept talking. I spoke for roughly half an hour, thinking through my different arguments, pausing where I wanted to pause and restarting twice, and ended up with a recording of everything that I had said and I would put in before, and ending up with recording, I would put it in one more like comma and then the words and adding notes like maybe cut that sentence or make sure this fact is correct, where appropriate. All right, back to the end of the piece. Then I fed this transcript, sorry. Then I fed this recording to a transcription tool, and copy and pasted the transcript into open AI’s 03 Pro, along with the following prompt, take this poor quality transcript of a first draft of a speech, complete with stutters, restarts and repetitions, and turn it into a single, well drafted, cohesive version of the same. Don’t change the structure, wording or argument unless absolutely necessary. Though, it would help if you could point out areas to tighten or clean up, check the numbers and make sure they’re correct, adding footnotes, footnotes with the math, where possible. My source for the spending data was link, and my source for the wealth tax impact was link, and I calculated the effect of the top 10% income tax, additional income tax in the top 1% as 3.3 million in parentheses, number of 1% in us by overall population, assuming that’s the denominator in the 800k cut off. End, parentheses, times 800,000 in parentheses, the approximate income cut off to be a one percenter, though I asked AI instead of having a solid source for that. End, parentheses, check my facts and make sure I didn’t say anything wrong, critique my arguments and suggest improvements or additional points where necessary. Finally, feel free to add anything else to the markup you think would be helpful. Here’s the transcript I got, and then I pasted the transcript. What I ended up with was my argument laid out in a structure much closer to how I wanted it to be, but still in need of some improvements for transparency. I’ll include the output of this first draft below, or maybe I would just say here’s the output of this first draft. At this point, I underwent revisions, mostly addressing the points that 03 pro had found, making them stronger where possible. I and correcting the argument to be the strongest version of itself. Finally, I added this to my personal website and hit Upload. The recording and transcription process to produce the original draft took around 45 minutes feeding it to 03 pro took another five and then I spent another hour or two on editing. I’m pretty happy with where this piece ended up, and I think it’s a much stronger version of where it originally was. And further, I find that this way of writing posts tends to stick to my original message far more closely than my previous way of writing it out manually. Not to mention that when you tell 03 pro not to correct your points or change your wording, it actually tends to lessen. I’ve read this entire piece end to end a few times at this point, and I really can identify any areas that aren’t the way that I would have said it you I’m probably going to stick with this way of writing going forward, if only because I’m so much more fluent and fluid while speaking out loud than while typing. If you’re curious, yes, this follow up was produced in much the same way I though, with a lot less editing, here’s the original transcript of the follow up that Is this very piece you’re Reading. If you’re curious.

Editor’s Note:

I was quite surprised reading the final draft above to find just how bad the transcription was. There were some major transcription mistakes (swapping “can” for “can’t”, missing words, truncated sentences, and several more), and yet the AI barely noticed. Truly amazing.

US Government Math

Mon, 07 Jul 2025 05:21:41 -0700

The US national debt is not a hard problem to explain. It’s actually a fairly easy problem to explain, but it’s going to take some tough solutions. The cause is that we’re spending a very large amount of money on exactly four things. Almost everything else in the budget doesn’t really matter.

One, health care

It used to be that if you were poor and got sick, you died, and people didn’t really care about the poor. Same goes for the elderly. When you get old, your body gets weak and you start having kinds of health problems you didn’t know were possible. Over the past few decades medicine has gotten dramatically more expensive, and in those same decades we’ve dramatically expanded the amount of health care we’re providing to the poorest and the oldest among us, to the poorest via Medicaid and to the oldest via Medicare. The problem is that in 2024 this cost roughly $1.8 trillion (Medicaid $912 B + Medicare $874 B)¹. I’m not saying we shouldn’t pay for this — we almost definitely should — but if you try to cut health care for the people who need it most, they tend not to like it.

Fifty years ago retirees were the single poorest demographic in the United States². There was an internal conflict in the mind of nearly every would-be retiree: With age, the body starts making living and working more difficult, but without the money to retire, at some point one would have to decide they can’t work anymore, and then try their best to live a healthy and hopefully happy life. We had let our elderly down. In the same way that Medicare fixed health care for the elderly, Social Security fixed poverty in the elderly by guaranteeing income in retirement.

Now the problem is threefold:

People live a lot longer than they used to; if you retire and die three years later you don’t need much Social Security, but if you retire and die twenty years later we have to support you for a quarter of your life.
We’re not putting enough money into the system to match what we pull out.
Roughly 73 million Baby Boomers (born 1946-1964) — the largest population boom in American history — are retiring, moving from peak productivity to drawing benefits³.

If we had accounted for this in advance, maybe we would have raised the amount of money you have to put into Social Security over time. As you can imagine, this is not a politically popular decision, and so we haven’t really done it. We never made it so that what you pull out is tied to what you put in; those are two entirely disconnected numbers in the law. Because of this, Social Security is, by most estimates, going to go insolvent in about eight years⁴, and then no more Social Security money, or more likely we’re going to need to print a lot more money. Social Security still costs about $1.46 trillion¹.

Three, interest

When you borrow money you have to pay it back, plus a little prize for the lender. Interest payments have been rising because the debt itself is rising and because we borrowed a lot during historically low rates. We originally had near-zero interest rates to help us recover from the Great Recession, slashed them back to near zero during the pandemic, and only now, while fighting the inflation that cheap borrowing helped create, have we raised them, at the cost of making all the borrowing we still have to do much more expensive. In 2024 we spent almost $882 billion on interest⁵, and as we push borrowing well past $30 trillion that number will climb every year, squeezing the economy as it does. This is what happened to Japan.

Back in the ’90s everyone thought Japan would lead the world economy. People were afraid of Japan’s economy the way they’re afraid of China’s now. Then it just stopped. Japan basically froze and has been frozen since the ’90s or early 2000s, largely because it borrowed huge amounts and now pours a huge share of its economy into paying interest. We are not Japan, and I don’t want us to become Japan.

Four, defense

Defense is hard to buy efficiently because there’s no competition. If General Mills triples the price of cereal you can buy Kellogg’s, but you can’t buy generic bombers. When the one company that makes the war-plane says it costs more, do you say, “That’s okay, we’ll just lose the next war”? No — you bite the bullet and spend it.

We just dropped 14 GBU-57 bunker-buster bombs on Iran’s nuclear program⁶. Developing the GBU-57 cost between $400 million and $500 million, and each bomb costs about $16 million⁶. A hundred of these bombs would cost us $1.6 billion⁶. Spending on groceries has declined dramatically over the past fifty years as a percentage of the average household’s spending, but do you know what hasn’t declined? The cost of defense. Continuing to buy bombs and war-planes is costing us roughly $874 billion a year¹.

There’s the rub

Health care, Social Security, interest, and defense together account for about 74 percent of the budget¹, and if we eliminated every other program tomorrow we’d still be borrowing money just to pay for these four things.

These four things are basically untouchable. You can’t cut health care for the poor or the elderly; you can’t cut Social Security because retirees are now a large, wealthy, and powerful voting bloc; you can’t stop paying interest without triggering financial Armageddon; and you can’t gut defense without risking military Armageddon.

The theory behind Doge was that we spend a substantial amount of the government money we spend on fraud. In truth, more than three-quarters of the dollars we spend are spent on things that are basically politically impossible to remove and logistically very difficult to reduce. Of Medicaid, Medicare, Social Security, interest, and defense, half of these if cut would earn you the fury of the elderly and the poor, both very substantial populations in the United States today, and the other two if cut would see the US face a collapse of the financial system or war.

If you want to understand how the Maslow’s hierarchy of American federal funding works, look no further than the fact that health care gets almost a trillion dollars a year, while transportation gets about $88 billion¹ and education gets only about $305 billion¹. You can live without transportation, but you can’t live without chemo, so of course the government spends a lot more on chemo.

Bonus, taxes

There’s an age-old idea that if only you properly taxed the wealthy, or immigrants, or big corporations, or whomever your least-favorite group is, then all these problems would go away. We’re currently borrowing about $1.833 trillion a year¹. Let’s see what a few of the most extreme tax increase proposals in recent memory would do to the deficit.

Suppose you added a new income-tax bracket for the 1 percent, ten percentage points higher than the current one. The top 1 percent starts around $682,577 in income⁷ and earns about 26 percent of all income⁸. This 10% jump would be a dramatic increase over the lower brackets, as the current jump to the highest bracket is 2% for taxpayers with incomes above $626,350 individual / $751,600 for married couples filing jointly, and then 3% for the next jump down. That surcharge would raise roughly $390 billion⁸, about one-fifth of the deficit.

Try a wealth tax instead. Senator Bernie Sanders’s plan to tax wealth over $21 million at 1 percent would raise about $260 billion a year⁹. Senator Elizabeth Warren’s 2-to-3 percent wealth tax on fortunes over $50 million would raise around $230 billion a year⁹. Helpful, but still nowhere near the deficit.

Perhaps increase corporate taxes by 50%. This would be as radical as any of the proposals above, and would be a fairly pronounced squeeze on small businesses, but with corporate income tax providing $494 billion in 2024, would yield an additional $247 billion¹⁰. Exclude every firm with less than $2.5 billion in assets, and you’re getting an additional $163.5 billion (50% of $327 billion)¹¹.

None of these policies alone comes close to closing the deficit. Fixing the US national debt will, by necessity, squeeze at least one very large group of Americans — the old via Social Security and Medicare, the poor via Medicaid, or everyone via higher taxes.

It turns out the math isn’t mathing because it’s really inconvenient math.

U.S. Treasury, Combined Statement of Receipts, Outlays, and Balances, FY 2024 outlay tables. https://www.fiscal.treasury.gov/files/reports-statements/combined-statement/cs2024/outlay.pdf ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎
U.S. Census Bureau, Historical Poverty Tables: People and Families, Table 7 — Poverty of People, by Age (1973–1975 columns). https://www.census.gov/data/tables/time-series/demo/income-poverty/historical-poverty-people.html ↩︎
U.S. Census Bureau, “The Baby Boom Cohort in the United States: 2020 to 2060” (P25-1149). https://www.census.gov/content/dam/Census/library/publications/2019/demo/p25-1149.pdf ↩︎
Social Security Administration, “Social Security Board of Trustees: Projection for Combined Trust Funds One Year Sooner Than Last Year” (press release, 2024). https://blog.ssa.gov/social-security-board-of-trustees-projection-for-combined-trust-funds-one-year-sooner-than-last-year/ ↩︎
U.S. Treasury, Combined Statement of Receipts, Outlays, and Balances, FY 2024—net-interest table. https://www.fiscal.treasury.gov/files/reports-statements/combined-statement/cs2024/outlay.pdf ↩︎
U.S. Department of Defense, “Historically Successful Strike on Iranian Nuclear Site Was 15 Years in the Making,” 24 Jun 2025. https://www.defense.gov/News/News-Stories/Article/Article/4227082/historically-successful-strike-on-iranian-nuclear-site-was-15-years-in-the-maki/ ↩︎ ↩︎ ↩︎
Investopedia, “How Much Income Puts You in the Top 1 %?” (IRS SOI 2021 threshold). https://www.investopedia.com/personal-finance/how-much-income-puts-you-top-1-5-10/ ↩︎
Tax Foundation, “Latest Federal Income Tax Data, 2024.” https://taxfoundation.org/data/all/federal/latest-federal-income-tax-data-2024/ ↩︎ ↩︎
Tax Foundation, “Details and Analysis of Senator Sanders’s Wealth-Tax Plan,” 2020 dynamic score. https://taxfoundation.org/sanders-wealth-tax-analysis/ ↩︎ ↩︎
U.S. Treasury, Combined Statement of Receipts, Outlays, and Balances, FY 2024 receipts tables. https://www.fiscal.treasury.gov/files/reports-statements/combined-statement/cs2024/receipts.pdf ↩︎
Internal Revenue Service, Statistics of Income Division, Corporation Income Tax Returns, 2021: Table 8—Income Tax After Credits, by Size of Total Assets. https://www.irs.gov/pub/irs-soi/21coccr.pdf ↩︎

AI and the Inversion of Control

Sat, 21 Jun 2025 19:17:16 -0700

Since SaaS¹ became a business model in the [mid 2000’s], the primary way by which companies integrate third party tools into their software has been via API. This is such a common and boring fact that most software people don’t give it much thought. Whether connecting a database, an analytics service, or sending notifications to a user’s phone, the languages is essentially the same: send an API request to a server somewhere, providing it whatever information it needs, and then process whatever you back in response.

The reason this is such an obvious and unavoidable interface is that databases, analytics services, and software to send notifications to a user’s phone are all tools. The buyer’s software chooses when to use them, and the software they bought patiently waits to be called. There is a second option, webhooks, used when you want your software to respond to external events. But despite the existence of a pattern for the software you buy to call your software via webhook, the use case is almost always NOT some tool messaging you and asking you to do something. As a rule, webhooks are used to inform your software that something has happened, and then it’s up to your software to decide what to do about it.

I write this in mid 2025, and we’re now beginning to see the emergence of a new class of software you can buy to integrate with your software, agents. Why does this matter? Because in order for an agent to be useful, it really helps if the agent can choose what it wants to do. What happens when an agent you pay for wants to check if a user received an email, or perhaps to send one from your service? What happens when a customer service agent you buy needs to exchange a user’s purchase for the next size up, or help them reset their password? When the agent wants to take an action that depends on your software, there are two ways of fulfilling its request:

delegate access to the underlying resource (e.g. email sending as your company) to the agent, or
create a way for the agent to ask your software to do something

In the former case, you might predict proxies to all the tools a company uses — or perhaps even the tools themselves — offering advanced authentication and authorization capabilities might become commonplace. This would necessitate the creation of software that doesn’t currently exist to enable this behavior.

In the latter case, this sounds a lot like a world where companies are going to be spending engineer hours building APIs into their software for the agents they buy to use, which sounds to me like it would scale terribly. One might imagine they’d address this by having coding agents write the endpoints instead of engineers, but that implies companies will need to tolerate dramatic increases in tech debt to support the needs of an ever increasing number of agents. I would predict there will need to be software to enable this too.

The implications of this are potentially the introduction of new channels for communication between buyers and sellers of SaaS. REST² APIs to tell your purchased software what to do, webhooks for your purchased software to tell you about things that happen in the world so you can react to them, and something new for agents to instruct your software to take action on their behalf.

I’m excited to learn what that thing will be.

SaaS, or Software as a Service, refers to the practice of buying software from a vendor who exists solely or primarily to sell this software to you, which you then either use via some web interface or app, or connect to your own software to use as described here ↩︎
Yes, REST has lost much of its meaning over the past decade or two. I’m referring to it (as most do these days) to describe the pattern of sending requests to software which does some work for you and then responds with a result ↩︎

Arguments About AI

Wed, 15 May 2024 07:45:04 -0400

Random Pertubations

Imagine M&Ms divided into four groups by color on a sheet of graph paper. The brown M&Ms sit on the positive side of the Y axis, the reds sit on the negative side of the Y axis, the blues sit on the positive side of the X axis, the greens sit on the negative side of the X axis. All other colors are absent.

This model represents any structured system. A human cell, a car engine, a software microservice. Pushing any of the M&Ms deeper into its own color group would increase the inherent structure, pushing any of the M&Ms further away from its color group would decrease it.

Most of the things we humans call “the world” depend on a large degree of structure to function properly. The most amazing devices — microchips with billions of individual transistors per square inch for example — tend to rely on the largest degree of structure.

If you were to randomly perturb an M&M, pushing it in any of the four possible directions on a 2D graph, one of those directions would push it deeper into its cluster, in the direction of increased structure, three would push it away from its cluster. Thus there is a 75% chance this maneuver would decrease the structure of the overall system.

The number “4” comes from the dimensionality of the system. The number of dimensions times two, in this case for a two-dimensional graph. The real world is far higher-dimensional. As a very rough proxy, CLIP embeddings, representing all possible images, have 1024 dimensions. In such a space, one direction pushes deeper into the vector for a given M&M color, 1023 push away, for a 99.902% chance of a random pertubation decreasing the structure of the system.

In our world, any powerful entity can push things around. When things are pushed, they are far more likely to be pushed in the direction of decreased structure than in the direction of increased structure. This is countered by the fact that the most important human decisions tend to be made with significant care, and these decisions don’t all have to be right. The higher dimensional the system, the more carefully the direction must be chosen to maintain or increase structure, as the lower the number of options that increase structure. When structure decreases in our world, it tends not to go well.

Autocompleting Regular Expressions

Wed, 08 May 2024 20:59:59 -0700

When designing forms to collect custom data types, you’ll sometimes have a regular expression for what you’re trying to collect, and for easy visual spacing you’d like to automatically inject separators into the form where appropriate, like this example for phone numbers:

>> user_input = "415"
>> phone_number = "\\d{3}-\\d{3}-\\d{4}"
>> append_separators(user_input, regex=phone_number)
'415-'

But how to implement append_separators, without losing the regular expression as the single source of truth? There are existing implementations for phone numbers and credit cards, but for company TIN numbers? Social security numbers? More exotic inputs? The closest Python libraries I’ve found to solving this are Exrex and Greenery.

Exrex allows for randomly sampling strings matching a given regular expression, hex color codes for example:

>> from exrex import getone
>> color_hex = "#(?:[0-9a-fA-F]{3}){1,2}"
>> getone(color_hex)
'#1D7EBb'

Greenery allows for applying logical operations to regular expressions:

>> from greenery import parse
>> five_plus_chars = ".{5,}"
>> str(parse(five_plus_chars) & parse(color_hex))
'#[0-9A-Fa-f]{6}'

Considering that Greenery works by converting regular expressions to finite state machines, computing the intersection of the two FSMs as a third FSM, and using the Brzozowski algebraic method (q.v.) to convert the third FSM back to a regular expression¹, re-implementing this to find where a separator can be appended would seem much higher lift than necessary to achieve the desired effect.

But if you can randomly sample from strings satisfying a regular expression:

>>> for _ in range(10):
...     print(getone(phone_number))
... 
237-178-8887
264-194-8260
945-873-3099
515-809-7348
112-255-4732
936-552-0686
422-686-3399
313-372-8544
309-291-8909
159-619-4738

Then you can see from the outputs which slots in the generated output are always the same. Would it be possible to solve this probabilistically?

# Generate 1000 strings satisfying this regular expression
options = []
for _ in range(1000):
    options.append(getone(phone_number))

# Find all characters that appear at each index
seen_by_index: dict[int, set] = {}
for option in options:
    for i, char in enumerate(option):
        # Add a new set if we don't yet have one for this index
        if i not in seen_by_index:
            seen_by_index[i] = set()

        seen_by_index[i].add(char)

        
# Print seen characters for each character position
for i, char_set in seen_by_index.keys():
    print(f"Character {i}: {sorted(char_set)}")

Output:

Character 0: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
Character 1: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
Character 2: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
Character 3: ['-']
Character 4: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
Character 5: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
Character 6: ['-']
Character 7: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
Character 8: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
Character 9: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
Character 10: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']

Let’s call a character position unique if there is only one character allowed in that position.

Given this is a probabilistic algorithm, let’s establish some bounds for error rates. Which error modes are possible? Type I error would be a character position appearing to be unique, while actually having additional possibilities which didn’t appear due to random chance. Type II error can be safely disregarded, as there’s no circumstance in which a truly unique character position would have other characters appear in a string without violating the regular expression. No other error mode is possible, so long as the regex doesn’t allow for variable-length strings, which we’ll address shortly.

The more characters are valid for a given character position, the less likely each is to appear, so the case making a false positive most likely is when there are the least number of possible characters for that position, specifically 2 characters, given 1 would make the character position truly unique.

If we flip a digital coin 100 times, the probability of only seeing one of two possible faces across all 100 flips — assuming both characters are equally likely — is 1/2^100, or:

0.000000000000000000000000000000789

With the benchmark set thus at 7.9e-31, or one false positive in 1.3e30 flips, if the entire human population of the Earth were to flip coins once a second for 14 billion years, or the approximate current age of the universe,² we’d have 3.5e27 flips, or a 0.3% chance of a single false positive after 14 billion years.

For some extra safety, we can flip 1000 times, for a false positive probability of 1/2^1000, or one false positive in 1e301 flips. At that point, the all-humans-every-second-since-the-big-bang flip count has no meaningful impact on the false positive probability, and an arbitrary precision calculator I used to test this just rounded it to 0.³

So for our complete algorithm:

from exrex import getone

class SeparatorAppender:
    def __init__(self, regex: str):
        self.regex = regex
        self.index = self._index_separators()

    def append_separators(self, user_input: str) -> str:
        to_append = self.index.get(len(user_input), "")
        return user_input + to_append

    def _index_separators(self) -> dict[int, str]:
        # Generate 1000 strings satisfying this regular expression
        options = []
        for _ in range(1000):
            options.append(getone(self.regex))
        
        # Find the allowed character set at each index
        seen_by_index: dict[int, set] = {}
        for option in options:
            for i, char in enumerate(option):
                # Add a new set if we don't yet have one for this index
                if i not in seen_by_index:
                    seen_by_index[i] = set()
        
                seen_by_index[i].add(char)
        
        # Find the index-to-separator mapping whenever only one separator is allowed
        separators_by_index = {
            i: list(charset)[0] for i, charset in seen_by_index.items() if len(charset) == 1
        }
        
        return separators_by_index
    

phone_number = "\\d{3}-\\d{3}-\\d{4}"
appender = SeparatorAppender(phone_number)
print(appender.append_separators("415"))

Output:

415-

Finally, handling regular expressions matching strings of varying length. For this, we can bring in Greenery, by simply merging the existing user input with the regular expression:

>> import re
>> user_input = "415"
>> user_input_as_regex = re.escape(user_input) + ".*"
>> str(parse(phone_number) & parse(user_input_as_regex))
'415-\\d{3}-\\d{4}'

If you can’t definitively say whether a character position is unique until after the character preceeding it is entered, then the corresponding character can’t be appended anyway, so by accounting for the existing user input, we’ve gone as far as we can without inserting symbols between already enterred characters.

import re
from exrex import getone
from greenery import parse

class SeparatorAppender:
    def __init__(self, regex: str):
        self.regex = regex

    def append_separators(self, user_input: str) -> str:
        index = self._index_separators(user_input)
        to_append = index.get(len(user_input), "")
        return user_input + to_append

    def _index_separators(self, user_input: str) -> dict[int, str]:
        # Combine the user input with the regex
        user_input_as_regex = parse(re.escape(user_input)) + parse(".*")
        regex = str(parse(self.regex) & user_input_as_regex)
        
        # Generate 1000 strings satisfying this regular expression
        options = []
        for _ in range(1000):
            options.append(getone(regex))
        
        # Find the allowed character set at each index
        seen_by_index: dict[int, set] = {}
        for option in options:
            for i, char in enumerate(option):
                # Add a new set if we don't yet have one for this index
                if i not in seen_by_index:
                    seen_by_index[i] = set()
        
                seen_by_index[i].add(char)
        
        # Find the index-to-separator mapping whenever only one separator is allowed
        separators_by_index = {
            i: list(charset)[0] for i, charset in seen_by_index.items() if len(charset) == 1
        }
        
        return separators_by_index


phone_number = "\\d{3}-\\d{3}-\\d{4}"
appender = SeparatorAppender(phone_number)
print(appender.append_separators("415"))

Output:

415-

Note that we unfortunately lose the ability to pre-compute our index for the regular expression and so there is some relatively significant performance cost to this approach. How much?

>> phone_number = "\\d{3}-\\d{3}-\\d{4}"
>> appender = SeparatorAppender(phone_number)   
>> %timeit appender.append_separators("415")
65.9 ms ± 3.78 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

And for the version without crossing in the existing user input to handle variable length inputs:

>> phone_number = "\\d{3}-\\d{3}-\\d{4}"
>> appender = SeparatorAppender(phone_number)
>> %timeit appender.append_separators("415")
75.9 ns ± 0.721 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

If you try reducing the number of rounds on the original regular expression to 100, tolerating that 0.3% chance of a single false positive in 14 billion years with the entire human population of Earth flipping a coin per second:

>> phone_number = "\\d{3}-\\d{3}-\\d{4}"
>> appender = SeparatorAppender(phone_number)
>> %timeit appender.append_separators("415")
47.9 ms ± 2.67 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Probably better to keep it to fixed-length strings.

This is actually an exact quote ↩︎
I say current age because by the time you read this the universe may be significantly older ↩︎
Really, this should be a count of false advertising against the “arbitrary precision” calculator ↩︎

Things I Would Pay For: Invariant Testing

Tue, 23 Apr 2024 12:22:03 -0400

I’m starting a new series on things I will pre-commit to paying for if someone builds it, or points me to one that meets my stated needs.

For the first run of the series, I spent several minutes looking through our API trying to guarantee there was no way a user could ever see data from a different organization, except if the data has been shared publicly. There are a few common solutions for this, like single-tenant systems, one tenant per organization, where data is copied to a “public” system once shared publicly by the user, but this is one of several similar invariants I need to guarantee. You can’t use tenancy as a knowledge boundary for every possible invariant.

You also can’t write tests asserting that something isn’t possible, for the same reasons it’s always hard to prove a negative. Fuzzing is an option, but the strength of the proven invariants tends to depend upon the skill of the person writing the fuzz tests.

The Build

I’d love to have a symbolic execution tool where I could write out invariants and it would prove whether or not they’re true, ideally as part of our standard suite of tests. Add in some AI for natural language writing of invariants à la “thou shalt not allow users to access other users’ data unless its visibility is set to public” or “thou shalt not allow users to assign tasks to other users they are not set as an owner for” and I could throw together a set of invariants in an hour or two covering all our basic security needs.

Aircraft manufacturers and similarly risk-intolerant companies (with the possible exception of Boeing) tend to use verification tools like TLA+ to specify the behavior of their systems, and guarantee through mathematical proof that there are no combinations of conditions that can result in behavior outside of specified constraints. The rest of us don’t use these tools because they take too much effort to set up and use. Using language models to bridge that gap would be a great way of turning natural language into value, and I’d personally pay some small amount every month to be able to eliminate these risks.

51 Percent

Thu, 21 Mar 2024 17:03:43 -0400

I often wonder what things I’m wrong about. I sometimes notice wrong things said by friends and colleagues, and I’m sure I’m not alone in that regard. I’d have to imagine at least 10% of what’s in my head is wrong, and that’s likely an optimistic estimate, but after seeing a sufficiently large number of wrong things published in prestigious places in a given week, I’m apt to wonder how this can be. Over the past few millennia, mankind has invented science, turned rocks into refined metal, sand into semiconductors, created intelligence in those metals and semiconductors, and sent men to the moon. Yet so frequently we’re wrong, and at first it seems contradictory.

Some of the smartest minds in physics spent the better part of fifty years studying string theory as a possible theory of everything, a single set of equations and principles that can predict with mathematical precision everything that does, did, and can ever happen in our universe. Only after decades of finding flaws, inventing complex rules to resolve those flaws, and finding more flaws in a loop did the house of cards crumble in the past few years. Many physicists bet their lives on it; many still do. How can so many brilliant people be led astray for so long?

I think the problem is in the question. The more fact-minded among us have a particular dislike for wrong things, and subconsciously see a world where all the important decisions being made by all the right people means society can keep moving forward. First, this notion is probably elitist. But more importantly, it’s probably wrong.

After spending a few hours watching a machine learning model train on a second monitor while doing my work for the umpteenth time, I realized my sense that the important things have to be right disagreed with the evidence presented by this intelligence being created before me. The model doesn’t take every step in the direction of self-improvement. If you were to watch it navigating a map in parameter space, many of its steps would be aimed in no particularly intelligent direction at all.

It doesn’t need to be right. It only needs to be just slightly more right than it is wrong, and it’s mathematically guaranteed to eventually converge on the optimum. This isn’t a figure of speech, it’s mathematical certainty, far greater mathematical certainty than string theory will every be: if your level of rightness is just slightly larger than your level of wrongness, you’re guaranteed to make progress. Society doesn’t need a world full of right people getting it right almost all the time; the mathematics of progress are fortunately far more forgiving. You don’t even need 51%. But of course, the more right you are, the faster you’ll converge.

Polyphyly and Other Cladistics

Fri, 12 Jan 2024 22:13:33 -0500

For today’s episode of “Really Important Concepts That Don’t Have Names,” I’d like to briefly introduce two words shamelessly stolen from the esoteric biological field of taxonomy.

Carl Linnaeus made his place in history through his system of organizing every material, living or dead, into species, branching from a taxonomic tree initially divided into the three classes of animal, plant, and mineral. Hardly the first to attempt such a thing — Aristotle’s system of classification had been used for about two millennia prior — Linnaean taxonomy would gradually incorporate discoveries from archaeological study of ancient species, study of newly discovered forms of ever smaller life, and eventually the discovery of Darwinian evolution.

As naturalists, now referred to as biologists,¹ sought to construct ever more accurate trees of life, a subtle change took hold in methodology. No longer was the scientific establishment satisfied with groupings derived solely from similarity with little regard for evolutionary history. This prior way of thinking, then referred to as phenetics, was gradually replaced by cladistics, the practice of categorizing species by shared ancestry.

But with genetic evidence and improvements in the ability to trace ancestral lineages of modern life, the field discovered a problem: sometimes, two species that appear to be very closely related are actually evolutionarily quite distant, and conversely, two species that appear to be quite different may actually be closely related.

The result is a number of taxonomic reorderings that I’m sure Wikipedia will suffice to describe, and two marvelous terms for a situation frequently encountered but poorly described for lack of terminology:

Polyphyletic, when two or more things may appear to be closely related, but actually come from different branches of the tree of life. Typically used to describe situations where two species appear very similar but are actually not very closely related.³

Paraphyletic, when two or more things may not appear to be closely related, but are actually quite closely related. Often used to describe situations where shared ancestry is responsible for an observed set of similar traits.

For a bonus vocab word, because these terms are primarily applicable to scenarios where groups of things share certain traits, those traits shared can be referred to as homoplasies.

So next time you find yourself in the middle of a discussion demanding more nuanced terms to distinguish between things belonging to classes as a result of shared ancestry versus shared traits, there’s a word for that.

A note for the taxonomists in the house, or those who crave their particular flavor of pedantry: as you might imagine, the degree of ancestral relation is of particular interest to those constructing trees of all life on the planet. Where does one draw the line between the polyphyletic and the paraphyletic? If a group of species includes all descendants of a given ancestor, it’s said to be monophyletic (which also happens to be the only one of these terms whose definition isn’t subject to debate). If it includes some but excludes others, it is said to be paraphyletic with respect to the excluded others. If a group is defined by a set of shared characteristics but those characteristics are not the result of shared ancestry, it is said to be polyphyletic.

Technically, the definition of paraphyletic I provided above also (incorrectly) includes monophyletic groupings, even though logically the two labels should be mutually exclusive. Given that the purpose of the vocab lesson is to introduce a concept that can be used to categorize ideas, and ideas, being not of a finite set, cannot be enumerated, monophylesis is practically a non-issue, and so I excluded the term above for simplicity. Feel free to contact me via my LinkedIn for a duel.

More accurately, naturalism split into biology and chemistry, the latter of which later split into chemistry and physics, likely with a number of additional fields I’m forgetting. ↩︎
Putting footnotes on footnotes is a rather underappreciated capacity of modern site rendering engines. I’ve taken the liberty of doing so for a fun if brief tangent on carcinization and horoscopes. You may have noticed the symbol for the horoscope cancer is a crab, and carcinoma is a type of cancer. As it happens, the word for cancer overlaps with the word for crab in a great many languages, with the most common explanations noting the hard surfaces and tight grip of tumors in the eyes of early surgeons, and the crab like appearance of the veins around a cancerous growth. Yet cancer again collides with crabs in the naming of carcinoma, from the Greek karkinos meaning crab or cancer, and -oma, which likely began its life as a suffix in the word carcinoma, but now means tumor or cancer. Two largely unrelated connections between crabs and cancer. Wild. ↩︎
The classic example of this — the exclusion of which may be the most common reason biology teachers are drawn and quartered in the modern day — is the rather absurd number of entirely unrelated species that look very much like crabs. The trend of evolving to be more like a crab is so common it’s even been given a name: carcinization.² ↩︎

Rules for Startups

Mon, 13 Nov 2023 15:01:16 -0500

1. Measure your progress, and be relentless about growing it over time.

You should probably use week-over-week revenue growth rate as your metric for this

2. Keep the customers happy, whatever it takes

If you can’t keep a given customer happy, maybe they shouldn’t be your customer

3. Things don’t change unless and until someone changes them

Either someone else changes something, or you do, or it stays the same

4. Execution > Directional Correctness > Ideation

Thinking of good ideas is easy, understanding the nature of the problem is hard, but execution outreturns both
The 51% rule implies that directional correctness is steering, execution is gas, ideation is really just picking landmarks

5. Ensuring people know about your product is just as hard as building a good product

Product decisionmaking and marketing/sales decisionmaking deserve equal attention, but are rarely given it

6. If time is the most precious commodity, then prioritization is the most important skill

7. Opportunities come in unusual packages, and rarely when you’re looking for them

Generally the most important steps toward a goal seem entirely random in the moment

8. Take breaks sometimes

Get good at noticing when your productivity is slipping and why, and if you’re going to rest, do it properly

9. No problem is below you

By default, there is no one else to take on problems you don’t want to solve, including ordering pizza and carrying trash
You can pay people to change this, but being able to delegate problems you don’t want to deal with comes with a price

10. Let others name things

Giving something its name is an unusually satisfying experience, but for most things it’s not actually that important
If you let others name things, they’ll feel they played an important role in its creation, and a missed opportunity to give an employee or colleague that feeling is rarely worth the joy of naming it yourself

How Do LLMs Do?

Tue, 01 Aug 2023 19:08:00 -0500

In conversations with founders, engineers, and enthusiasts of ChatGPT-enabled academic dishonesty, one point of confusion about LLMs that frequently surfaces is how, on a technical level, services that give LLMs access to information from databases, Wikipedia, or even the Weather Channel actually work. How do you take an AI-powered chatbot that takes in words and spits out words and give it access to the internet, databases, or anything else of personal or economic interest? Over the next few thousand words, let’s break this down. This post is intended to be accessible to a non-technical audience. We’ll be using the terms prompt¹, tokens², training³, and fine-tuning⁴. If you’d like a definition for any of those, follow the little numbers.

Summary

There are only two possible ways for an LLM to know anything:

It’s in the training data
It’s in the prompt

Alright, thanks for reading. See you next week.

What?

Every time any LLM-based product answers questions, or generates output using any information at all, the software behind the scenes is using one or both of the above two strategies to give the LLM access to that information. I’ll explain the most common techniques for both by way of a few scenarios, but first, an aside on how LLMs are trained that’ll be relevant later on.

Training

The most common way an LLM is trained involves four steps. A useful metaphor is that an LLM’s “brain” consists of a fixed structure, and a rather long collection of numbers (“parameters”) that slot into that structure as everything the brain knows and can do, almost like downloading your brain as a long list of numbers. These numbers are adjusted directly to “train” the brain. GPT-3.5-turbo, one of the two LLMs behind ChatGPT, is expected to have 154 billion adjustable parameters.

Take your text data, and generate pairs of prompts and desired responses from the LLM for each, like so:

Input: "What's the square root of 4?"
Output: "The square root of 4 is 2"
Input: "Why can't humans fly?"
Output: "Human chest muscles are too small and weak relative..."
Input: "Great to meet you!"
Output: "It's great to meet you too!"
Input: "Who are you"
Output: "I'm ChatGPT, an AI chatbot built to help you with various..."
...

Feed each input to the LLM, and measure how different the LLM’s output is from the desired output in the training data
Use math to figure out which parameters in the LLM’s “brain” are responsible for most of the difference between the desired output and the actual output of the LLM, and then add or subtract just a bit from each parameter to try to correct this error
Repeat with more data

In the process of adjusting the LLM’s parameters to close the gap between the output it gives for a particular input and the desired output, the math behind LLM training ends up encoding a version of the patterns within the training data in its parameters. How facts translate into LLM parameters is poorly understood and an active area of study, but suffice it to say that your brain does something similar in encoding information within its equivalent to parameters.

The Executor

The key to the whole system is a piece of software that interfaces between the natural language input and output of the LLM and the rigid API’s of the software world. If there exists a way with code to turn on some lights, or fetch some data, or to do anything else useful, the LLM can’t do it by itself. It needs a layer of software that can translate the LLM’s output into a series of API calls, and translate the API’s response into a response the LLM can understand.

Examples

“ChatGPT, when is William Shakespeare’s birthday?”

This one is all “strategy 1” above. When OpenAI trained GPT-3.5-turbo and GPT-4 (the LLMs underlying the ChatGPT product), somewhere in all the training data was at least one reference to Shakespeare’s birthday. When the LLM was trained, at least one of the input-output pairs in the training data mentioned the date of his birth, and the resulting adjustments to the LLM’s parameters encoded this information within the LLM.

“Siri, what’s Johnny Appleseed’s phone number?”

While Siri isn’t yet LLM based as of the time of this writing, it almost certain will be very soon. Here’s how it’ll answer the above question.

(Behind the scenes)

Who	What do they say?
User (to Siri)	Siri, what’s Johnny Appleseed’s phone number?
Executor	Hey LLM, a user asked the following question: “Siri, what’s Johnny Appleseed’s phone number?” You’ll likely need to use a tool to answer it, so here’s a list of tools I can use for you. If you want to use a tool, put the name of the tool in brackets (e.g. {Calculator}), otherwise simply respond with plain text to reply directly to the user. Do not use brackets anywhere else in your output: - Calculator - Calendar - Contacts - Notes - Weather …
LLM	{Contacts}
Executor	I see you want to use contacts. You can take any of the following actions. To pick an action, put its name in brackets (e.g. {Add Contact}), otherwise simply respond with plain text to reply directly to the user. To go back, use the {Go Back} action. Do not use brackets anywhere else in your output: - Add Contact - Edit Contact - Delete Contact - Retrieve Contact … - Go Back
LLM	{Retrieve Contact}
Executor	To find a contact, put some search text in brackets (e.g. {Philip}), and I’ll give you a list of matching contacts to choose from. Otherwise simply respond with plain text to reply directly to the user. To go back, search for {Go Back}. Do not use brackets anywhere else in your output
LLM	{Johnny Appleseed}
Executor	Here are the matching contacts. To pick one, put its number in brackets. To get the next 10 contacts, respond with {Next}, otherwise simply respond with plain text to reply directly to the user. To go back, search for {Go Back}. Do not use brackets anywhere else in your output: 1. Tommy Baynard 2. Bobby Parsnip 3. Danny DeVito 4. Sammy Solomon 5. Ricky Pinkus 6. Johnny Appleseed 7. Joey Barbossa 8. Billy Taylor 9. Frankie Fortress 10. Tommy Applefarmer
LLM	{Johnny Appleseed}
Executor	Invalid Input. Here are the matching contacts. To pick one, put its number in brackets. To get the next 10 contacts, respond with {Next}, otherwise simply respond with plain text to reply directly to the user. To go back, search for {Go Back}. Do not use brackets anywhere else in your output: 1. Tommy Baynard 2. Bobby Parsnip 3. Danny DeVito 4. Sammy Solomon 5. Ricky Pinkus 6. Johnny Appleseed 7. Joey Barbossa 8. Billy Taylor 9. Frankie Fortress 10. Tommy Applefarmer
LLM	{6}
Executor	Here is the requested contact. To go back, respond with {Go Back}, otherwise simply respond with plain text to reply directly to the user. Do not use brackets anywhere else in your input: Name: Johnny Appleseed Phone: (123) 456 7890 Address: 1 Min Strot Lane, Arlington, GA 39813
LLM	(123) 456 7890
Executor	Just to confirm, you put the following in plain text: “(123) 456 7890” Are you sure you want to say this to the user? Say {Yes} or {No}, or respond with {Go Back} to go back. Otherwise, simply respond with plain text to reply directly to the user. Do not use brackets anywhere else in your input.
LLM	{Yes}
Siri (to User)	(123) 456 7890

You may notice the similarity between the above interaction and working through a phone tree. It turns out this is actually how we give access to tools: we have a software-defined phone tree (sometimes referred to as the “executor”), and the LLM takes the role of “user” talking to it. The software administering the phone tree, like real-world phone trees, just does dumb text matching. It checks at each step if the result from the LLM is in its list of available options, and has a number of hardcoded functions that actually do whatever the LLM asks for.

The executor is ultimately responsible for interacting with the outside world on behalf of the LLM; and the LLM is responsible for choosing from the options provided at each step to produce the desired result. The biggest difference is, this entire process can take place in seconds or less, so to the user it can feel nearly instantaneous. LLM’s are also less likely than most humans to get frustrated by the process, and most are quite forgiving of being put on hold.

“Tesla, turn on the AC”

This is where we start to see the real power of LLMs put to work. The executor need not only read information, it can execute any function requested by the LLM, provided it has a function to run. The executor is responsible for taking the LLM’s output and translating it into a command that the AC can understand, and the LLM is responsible for choosing from the options provided at each step to produce the desired result.

(Behind the scenes)

Who	What do they say?
User (to Car)	Tesla, turn on the AC
Executor	Hey LLM, a user asked the following question: “Tesla, turn on the AC” Pick a subsystem or resource to respond to this command from the following list. To select a subsystem or resource, put its name in brackets (e.g. {Suspension Control}), otherwise simply respond with plain text to reply directly to the user. Do not use brackets anywhere else in your output: - Engine Control - Transmission Control - Brake/Traction System - Suspension Control - Telemetry - Climate Control - Infotainment System - Cabin Controls - Internet Search
LLM	{Climate Control}
Executor	I see you want to use climate control. You can take any of the following actions. To pick an action, put its name in brackets (e.g. {Set Temperature}), otherwise simply respond with plain text to reply directly to the user. To go back, use the {Go Back} action. Do not use brackets anywhere else in your output: - Set Temperature - Set Fan Speed - Set Mode - Set Airflow Direction - Set Airflow Recirculation - Set Airflow Defrost - Set Airflow Rear Defrost … - Go Back
LLM	{Set Mode}
Executor	To set climate control mode, choose from the following options. To pick an option, put its name in brackets (e.g. {Fan Only}), otherwise simply respond with plain text to reply directly to the user. To go back, use the {Go Back} action. Do not use brackets anywhere else in your output: - Heating - Cooling - Fan Only - Go Back
LLM	{Cooling}
Executor	Just to confirm, you would like to change the climate control mode from “Fan Only” to “Cooling”? Say {Yes} or {No}, or respond with {Go Back} to go back. Do not use brackets anywhere else in your input.
LLM	{Yes}
Car	AC Turns On

Any action that can be expressed as a function can be executed by the executor, and the LLM can be trained to choose from a list of options provided by the executor to produce the desired result. This is the core of how LLMs can be used to control any system that can be controlled by a computer.

Caveats

As you can imagine, this is a slightly oversimplified depiction of things, but not by much. While human-facing phone trees tend to be optimized to minimize labor cost for customer service agents, and then perhaps frustration if the provider is feeling generous, LLM-facing phone trees are optimized around a different set of constraints, roughly:

Minimizing risk of making a mistake that could upset a user or cost the company money or reputational damage
Minimizing the compute cost for the LLM, which in turn means minimizing the amount of text input to and output from the LLM

Less important but still relevant:

Minimizing the number of round trips between the LLM and the phone tree
Maximizing the amount of useful data that can be acquired from this process

Nearly every element of this system can be optimized in myriad ways to meet these aims. There are papers describing innovations from replacing full-text responses to the phone tree ({Retrieve Contact}) with random tokens ({Q}) to save tokens and decrease bias⁵, to incorporating caching processes that allow for intelligently skipping steps that were recently performed and cached, like so:

Who	What do they say?
User (to Siri)	Siri, what’s Johnny Appleseed’s phone number?
Executor	Hey LLM, a user asked the following question: “Siri, what’s Johnny Appleseed’s phone number?” You’ll likely need to use a tool to answer it, so here’s a list of tools I can use for you. If you want to use a tool, put the name of the tool in brackets (e.g. {Calculator}), otherwise simply respond with plain text to reply directly to the user. Do not use brackets anywhere else in your output: - Calculator - Calendar - Contacts - Notes - Weather …
Executor Helper	By the way LLM, this sounds very similar to a previous user request, “Siri, can you get me Jimmy Robbins’ number?”, resolved with the following steps: 1. Choose a tool → {Contacts} 2. Choose an action → {Retrieve Contact} 3. Provide argument for action → {Jimmy Robbins} 4. Choose from results → {1} 5. Output to User → (402) 322 2748 6. Confirm → {Yes} If you’d like to skip ahead to just after one of these steps, put the number of the step in brackets (e.g. {4} for “Choose from results”). Otherwise, continue following the instructions above.
LLM	{2}
Executor	To find a contact, put some search text in brackets (e.g. {Philip}), and I’ll give you a list of matching contacts to choose from. Otherwise simply respond with plain text to reply directly to the user. To go back, search for {Go Back}. Do not use brackets anywhere else in your output
LLM	{Johnny Appleseed}
…

The above is a relatively simple example of a caching process, but it’s not hard to imagine more sophisticated ones that can cleverly reuse previous lookups and produce intelligently rewritten full conversation histories from cached previous interactions. While it’s still early days for this type of tooling, this is a fairly active area of research, and we’re likely to see a lot of progress in the coming months and years.

Final Questions

What if the LLM makes a mistake?

That depends on whether the LLM feeds an invalid response to the phone tree (like replying {Johnny Appleseed} for the contact above when the phone tree was looking for a number from 1 to 10), or a valid but incorrect response to the phone tree (like replying {5} instead of {6}).

If the response is invalid, in the most basic case the phone tree simply asks again, but more advanced implementations can use the same types of tricks that human phone trees go through, like “Did you mean…” as we’ve already got software that does that.

If the response is valid but incorrect, the best defense is ample access to “Go Back” and asking for confirmation before doing anything that could make a user unhappy, like deleting a contact or spitting out the wrong number. You can see this in the last few steps above.

But how did the LLM know the answer at the end?

When you feed text to an LLM, you don’t feed it just the most recent message, you feed it the entire conversation, or some version of it. The LLM has access to anything you include in the prompt, and its job is simply to generate the next message. So the LLM can see the answer just by looking at the rest of the conversation in its input.

What if the user’s request involves multiple steps?

In the simplest case, the LLM is clever enough to know to say “{Go Back}” a few times at the end to start over with a different tool, and as mentioned above, it can see everything prior in the conversation. As long as all the information it needs is somewhere in the conversation, the LLM has a good shot at finding it and using it correctly.

In more complicated cases, the phone tree might first ask the LLM to construct a strategy, and then perform some degree of handholding to walk the LLM through the steps in its strategy. For situations that require even more demanding logical flows, a number of logical frameworks exist to allow the LLM itself to take control of the process of arriving at the desired result, rather than letting the phone tree dictate the flow. Self-Ask and ReAct (not to be confused with the very popular frontend framework of the same name) are two well-known frameworks in this category.

Here’s what a ReAct phone tree looks like, for the curious. Note how the Executor finishes its messages with an Observation and the start of a Thought, and the LLM picks up where it left off, completing the Thought and then providing the next Action to take.

Who	What do they say
User (to LLM ReAct Agent)	Who is the current president of France, and what is the last letter of his birth month?
ReAct Executor	Answer the following questions as best you can. You have access to the following tools: google_serper: A low-cost Google Search API.Useful for when you need to answer questions about current events.Input should be a search query. Calculator: Useful for when you need to answer questions about math. Use the following format: Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, should be one of [google_serper, Calculator] Action Input: the input to the action Observation: the result of the action … (this Thought/Action/Action Input/Observation can repeat N times) Thought: I now know the final answer Final Answer: the final answer to the original input question Begin! Question: Who is the current president of France, and what is the last letter of his birth month? Thought:
LLM	I need to look up the current president of France Action: google_serper Action Input: current president of France
ReAct Executor	Observation: Emmanuel Macron Thought:
LLM	I need to determine the last letter of his birth month Action: google_serper Action Input: Emmanuel Macron birth month
ReAct Executor	Observation: December Thought:
LLM	I now know the final answer Final Answer: Emmanuel Macron, December - r
Result (to User)	Emmanuel Macron, December - r

What about OpenAI’s Function Calling API?

Function calling is a framework for reducing the amount of work needed to build the executor. It provides the LLM with a list of “functions”, or specific actions the LLM can take, and then has the LLM choose from that list and provide whatever inputs are necessary to run the function. It’s then up to the user to actually execute the function. It’s a relatively simple framework, telling the engineer that the LLM requested to use function check_weather and leaving it to the engineer to identify and run the correct code to check the weather given the LLM’s inputs. There exist a number of alternative frameworks that provide more or less of the planning and execution sides of the executor, but they’re mostly just different flavors of phone tree.

Prompt: The text fed into an LLM as input ↩︎
Tokens: The smallest, most granular unit of text you can feed an LLM. For technical but mostly irrelevant reasons, LLMs generally enforce a limit to the number of characters you can feed them for a single request, measured in tokens. Each token corresponds to around 4 characters. For the curious, see here ↩︎
Training: The process of taking a LLM (or any machine learning model really) and adjusting its “brain” to push how it’ll respond to a given input in a desired direction ↩︎
Fine-tuning: Doing additional training on an already trained model ↩︎
This is called “symbol tuning” and is described further in this paper. ↩︎

Posts on Tal Globus

Expect It to Fail

Doing Something That’s Never Been Done Before

A Follow-up to US Government Math

Unedited transcript (verbatim)

Editor’s Note:

US Government Math

One, health care

Two, Social Security

Three, interest

Four, defense

There’s the rub

Bonus, taxes

AI and the Inversion of Control

Categories

Arguments About AI

Random Pertubations

Autocompleting Regular Expressions

Things I Would Pay For: Invariant Testing

The Build

51 Percent

Polyphyly and Other Cladistics

Rules for Startups

1. Measure your progress, and be relentless about growing it over time.

2. Keep the customers happy, whatever it takes

3. Things don’t change unless and until someone changes them

4. Execution > Directional Correctness > Ideation

5. Ensuring people know about your product is just as hard as building a good product

6. If time is the most precious commodity, then prioritization is the most important skill

7. Opportunities come in unusual packages, and rarely when you’re looking for them

8. Take breaks sometimes

9. No problem is below you

10. Let others name things

How Do LLMs Do?

Summary

What?

Training

The Executor

Examples

“ChatGPT, when is William Shakespeare’s birthday?”

“Siri, what’s Johnny Appleseed’s phone number?”

“Tesla, turn on the AC”

Caveats

Final Questions

What if the LLM makes a mistake?

But how did the LLM know the answer at the end?

What if the user’s request involves multiple steps?

What about OpenAI’s Function Calling API?