AI Agents Are Terrible Freelance Workers

AI is getting better at rote tasks, but it still takes a human to do 97% of typical freelance tasks (according to research published by a company that sells the services of humans who tag material to train LLMs).

Last year I went through part of the training process for a job that offered “up to” $90/hr to PhDs who would write responses to prompts and/or rate responses (that might be written by humans, might be written by AI). As far as I could tell, I was supposed to go through the training on my own time, at no pay, and then sit around hoping for work.

I put in a few hours because I was curious, but stopped once I realized what kind of intellectual tasks the tech barons thought was still worth paying humans to do, and then I spent the summer re-thinking my teaching in order to emphasize what humans can still do better than algorithms.

This Wired story is not related to my personal experience, but it does show that GenAI chokes when it comes to messy real-world tasks that involve a completed model, general instructions, and a folder of raw files. When the workflow involves using multiple tools and, humans are still much better than clankers.

The researchers gave several leading AI agents a range of simulated freelance work and found that even the best could perform less than 3 percent of the work, earning $1,810 out of a possible $143,991.

[…]

Spectacular AI advances have led to speculation about AI soon surpassing human intelligence and replacing vast numbers of workers. In March, Dario Amodei, CEO of Anthropic, suggested that 90 percent of coding work would be automated within a matter of months.

Previous waves of AI have inspired misplaced predictions about job displacement, for example concerning the imminent replacement of radiologists with AI algorithms.

The researchers generated a range of freelance tasks through verified Upwork workers. The tasks span a range of work including graphic design, video editing, game development, and administrative chores like scraping data. They combined a description of each job with a directory of files needed to perform the work and an example of a finished project produced by a human.

Hendrycks says that while AI models have gotten better at coding, math, and logical reasoning in recent years, they still struggle to use different tools and to perform complex tasks that involve numerous steps. “They don’t have long-term memory storage and can’t do continual learning from experiences. They can’t pick up skills on the job like humans,” he says. —Wired