What is an AI employee? What you're actually buying

A company selling an AI employee has no reason to lead with what the product can’t do. This is the version built from primary documents instead: the engineering guidance from a company that makes the underlying models, and independent measurements of what AI agents can actually finish.

One disclosure first. Our own AI agents page uses the line “AI employees that never sleep.” It does run around the clock. Employee is shorthand, and this post explains what’s behind it.

What it is underneath

An AI employee is a language model, the same kind of system behind ChatGPT or Claude, connected to some of your tools (email, a calendar, a CRM, a help desk) and given instructions for a job.

Anthropic, the company that makes Claude, draws a useful line in its engineering guide, Building effective agents. It calls the whole category “agentic systems” and splits it in two:

Workflows are systems where LLMs and tools are orchestrated through predefined code paths.

Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.

Most products sold as AI employees are one of these, or a mix of both, with a name and a job title on top. The difference matters when you’re buying. A workflow runs the same steps every time, so you can predict what it will do and check its work. An agent chooses its own steps, which lets it handle messier situations and also lets it go wrong in ways nobody planned for.

Anthropic’s own advice to developers leans toward the simpler option: “When building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed.” On agents specifically, the same guide says: “The autonomous nature of agents means higher costs, and the potential for compounding errors.” That advice comes from the company that sells the models.

What the measurements say it can do

METR is a research nonprofit that measures what AI systems can do. One of its main measurements is how long a task an AI agent can complete with 50% reliability, expressed as how long the same task takes a skilled person.

In March 2025, METR reported that “current models have almost 100% success rate on tasks taking humans less than 4 minutes, but succeed <10% of the time on tasks taking more than around 4 hours.” It also found that the length of task agents could complete with 50% reliability had been doubling roughly every seven months for six years. The numbers have moved since then. METR keeps a live chart on its time horizons page, and that’s where to look for current figures, which are measured mostly on software tasks.

For anyone thinking about handing real work to one of these systems, three details on that page matter more than the headline number.

The test tasks are cleaner than real work. In METR’s words:

Our tasks are designed to be self-contained and well-specified, so that they’re fair to both the AI agents and the humans. In contrast, most real-world work draws on prior context, such as previous conversations, tacit knowledge, or familiarity with an existing code base.

METR’s own comparison is a new hire:

We think it’s better to think of our 2-hour tasks as what someone with low or no prior context (like a new hire or freelance contractor) could complete in 2 hours, rather than someone experienced who is already familiar with the project.

Reliability is uneven even among tasks of the same length. METR gives an example: on tasks that take a human expert 90 minutes to 3 hours, a GPT-5 agent “succeeds 100% of the time for around one-third of the tasks, fails 100% of the time for around one-third of the tasks, and sometimes succeeds and sometimes fails on the remaining third of tasks.”

Put together, the fair picture of an AI employee is a capable new hire on their first day. It works quickly and around the clock, and it does well on short tasks that are clearly defined. What it doesn’t have is what a person picks up by being around your business, like which customers need extra care or why the last quote was discounted.

What businesses are actually seeing

Two recent studies from the National Bureau of Economic Research show where businesses actually are.

In Firm Data on AI, researchers surveyed nearly 6,000 senior executives in the US, UK, Germany, and Australia. They found that “69% of firms actively use AI,” and also that “executives report little own-firm impact of AI over the last 3 years, with nine-in-ten reporting no impact on employment or productivity.” The 69% is much higher than the Census Bureau’s figure for US businesses, between 17% and 20%, and it comes from a different survey asking a different question. Looking ahead, the same executives predicted AI would raise productivity at their firms by an average of 1.4% and cut employment by 0.7% over the next three years. The paper contrasts that with employees, who anticipate AI will raise employment at their firms by 0.5% over the same three years.

In The Microstructure of AI Diffusion, researchers working with Census Bureau survey data found that among firms using AI, “Most firms (66%) use AI for task augmentation, while employment reductions are rare (2%).”

So in practice, businesses that use AI mostly use it to help people get tasks done. That’s a long way from the replacement hire the word employee suggests.

Where one earns its keep

The pattern from the research lines up with what we see building these systems. The jobs worth handing over are short, repetitive, clearly defined, and easy to check:

• The first reply to a new inquiry, including the two or three qualifying questions you always ask.
• Booking an appointment into your real calendar.
• Logging a new lead in the CRM with the details filled in.
• Summarizing a call or an email thread into the customer record.
• Drafting a reply for a person to review and send.
• Sending follow-ups on a schedule so nobody has to remember them.

The jobs to keep with people are the ones that depend on history or judgment: pricing exceptions, complaints, negotiations, anything involving a promise, and any decision that’s expensive to reverse. A useful test is whether you’d let a new hire do it unsupervised in their first week.

We covered the broader list of what’s realistic in what AI can actually automate in a business, and the jobs worth starting with in AI agents for small business.

How to judge one before you buy

Most demos show the product working. These questions get at what happens the rest of the time:

1. Is it a fixed workflow or an agent that picks its own steps? Either can be the right choice. You should know which one you’re getting.
2. Which of your systems can it change, and which can it only read? Anything it can change, it can also get wrong.
3. What does it do when it doesn’t know the answer? You want it to hand off to a person.
4. Can you read every conversation and every action it took? Without a log, there’s no way to check what it told a customer.
5. Who updates it when your prices, hours, or policies change, and how quickly?
6. What does it cost at your volume, including the AI usage fees behind it?
7. Can you trial it on one job for 30 to 60 days against a number you measured beforehand? Pick one number to measure, such as response time or hours saved. Without a baseline, you’ll be judging it on how the demo felt.

It’s also worth asking the vendor to show you what happens when it gets something wrong.

If you want one built around a specific job in your business, with the limits written down and every conversation reviewable, that’s the work on our AI agents page.