Dive Brief:
- The ability to direct advanced artificial intelligence systems is becoming increasingly important for early-career workers, according to a joint field study from the McCombs School of Business at the University of Texas at Austin and accounting firm KPMG.
- The research found that “nearly identical knowledge and skill can produce dramatically different results” among junior employees who work with AI agents. However, workers who excelled with AI “treated AI as a collaborator that needed direction, oversight, and judgment,” rather than simply delegating tasks to it.
- Underperforming workers offered AI critiques that didn’t improve on what AI delivered on its own. Instead, they often ended up “chasing irrelevant issues or steering the AI the wrong way,” per the report.
Dive Insight:
The study examined more than 523 early-career professionals at KPMG by asking them to use AI agents to complete workplace tasks. After setting a baseline, researchers tested employees to determine what separated the employees who improved AI’s output from those who matched it and the ones who fell behind.
According to the study, the employees who fell short “scored just as high on critical thinking, domain knowledge, and AI literacy as the amplifiers who pulled ahead.” Researchers concluded that the primary factor separating them “was how they worked with AI, not what they knew.”
“We weren’t simply looking for people who knew how to use AI,” Ashish Agarwal, professor at The University of Texas at Austin and co-author of the study, said in a statement. “We wanted to understand what enables some individuals to consistently create value beyond what AI can produce on its own.”
Researchers recommended that companies reconsider what kind of value their employees can add to the AI systems they’ve integrated. As more advanced AI models generate useful deliverables on their own, new roles will need to be developed to direct, evaluate and extend these results. Organizations need to make these kinds of assessments visible by having people explain why they made certain decisions, and “document why they accepted, changed, or rejected AI's output.” By doing this, researchers said companies can turn “an invisible skill into a coachable one; and grade the process, not the deliverable.”
“This is the most AI-native generation entering the workforce, so if fluency with the tools isn’t what sets the top performers apart, that tells us something about our entire workforce,” Rahsaan Shears, AI enterprise transformation leader at KPMG US, said in a statement. “How people applied their knowledge and skill is what made the difference, and that gap is coachable. The opportunity for organizations is to build the training and workflows that enable far more people to turn their knowledge and skill into impact, at every level.”
Making the assessment process more transparent may also help to identify critical skill gaps. Earlier this month, employee training platform TalentLMS released a report that found that many workers are using AI to cover up for a lack of knowledge. Employees who use AI to finish work they wouldn't otherwise be able to do aren’t developing the skills they will need to advance, per the report.
In addition, checking and correcting bad AI work is taking up a lot of time, according to a report recently released by Glean. That study found that digital employees spend nearly one full workday a week “botsitting” in order to make AI output usable, including fixing “confident-but-wrong” answers.
