(Essentially) free verification of facts and skills

Author: James Brand
Date: 2026-01-02

1. I’ve been thinking about this post for weeks and struggling to make my thoughts coherent. After seeing Tom Cunningham’s post on very related ideas finally decided to put something down, however imperfect. YMMV!
2. I say “skill” and “facts” without defining them below. Mostly, that doesn’t matter for the points.

TL;DR: AI is going to make it much easier to sort low- from high-skill workers at scale. This will increase the returns to those skills and open new markets for pre-emptive, re-usable, skill verification.

One of the biggest challenges in my experience doing technical work with data in a business setting is that, at some point, my work has to land on the desk of someone who doesn’t understand what I do and, through some combination of my explanations and pure trust (in me, those who hired me, or the HR system in whole), they have to be convinced that what I’ve done is worth listening to and/or acting on. The financial risk at that decision point is huge – when this is happening in the tech industry, every single decision is probbaly “worth” (in terms of the incremental value of getting it right) many millions of dollars.

The fundamentals of this dynamic hold in other fields. A product manager is presenting about the need for a new feature – have they done the right due diligence? Are the facts they’re presenting correct? The financial analysis? As a result, technical hiring is a super high-risk endeavor and becomes more so the higher up the corporate ladder we’re talking about. Unfortunately, reducing this risk is hard in practice. Mistakes in any sufficiently complex technical output are hard to catch, let alone catch in a 1-hour interview, meaning that distinguishing top 10% candidates from top 1% candidates is as difficult as it is critical to get right, especially as top technical talent continues to recieve amazing compensation amounts from e.g. OpenAI, Meta, etc.. Imagine paying someone $1M+ per year and then realizing they aren’t the absolute best at what they do – I think this happens more than those outside the industry might realize.

This is how I’d summarize the current challenge in identifying technical superstars, or even just “top” talent: 1. Evaluating skill is hard and high-noise, and it is nearly impossible to evaluate someone who is more skilled than you are. 2. The top (1%, 0.1%, 0.01%) of the talent/knowledge/skill distribution is far more valuable to businesses than even slightly weaker candidates. - Being vague about “skill” bites a little here because in some ways this might be tautological, but I’m trying to say that the distribution of dollar-value of skill has a really long tail on the high end. See endless discussions of “10x engineers” online for anecdotal agreements.

The graph below summarizes my thinking (don’t think too hard about the exponential shape of the blue line – if anything, I think it might be flat or negative for the first 50% of the distibution and steeper for the top 1%)

Hopefully the mismatch is obvious then – just as we get to the most impactful skill levels, we find that we don’t have anyone who can evaluate those skills!

Current technical hiring is bad, but AI will help

I don’t know how hiring for roles outside of data science go, so I’m focusing on DS here. In my limited experience, the process at big tech companies for hiring data science talent is pretty uninspired. A few hour-long interviews with someone at or above the hiring level, often with some mix of behavioral and technical questions. In some places, the technical questions are broad case studies chosen by interviewers, and in others they include a standardized technical quiz. Pros and cons to both, and I think they key trade-off is something like adaptiveness (case studies) vs. standardization (quizzes). Standardized questions are great for filtering out people below some broad minimal threshold, but are unlikely to detect superstars, and case studies will catch top talent on a few dimensions but are more likely to miss it on others.

In a constraint-free world with endless interviewer time and patience, both of these would be awful ways to judge talent – particularly at the top of the distribution, where big tech nominally wants to hire from. Noise in this search process, paired with my plot above, results in all kinds of dynamic effects. Low-skill candidates slip through the screen, get promoted based on bad (but difficult to verify) work, and then become hiring screeners for the next round of candidates, increasing the noise further and making it even harder to find top talent.

Not to say that there are no good DS teams out there – there are many – but there is a severe cold start problem, and the noise in the process remains.

AI makes two important verification steps btter

1. AI integration into meeting software (Teams, Zoom) makes it much easier to fact check live and ask better technical questions.
   • “Is this number right? Find internal docs supporting or contradicting this claim.”
   • “Explain this technical concept more clearly.”
2. Verifying workers’ skill up front is much lower cost, because an LLM can conduct an adaptive interview to

I mention (1) mostly because I think that’s what we’re going to see proliferate first. This is related to Tom’s post about the verification checklist. In some ways, I could imagine executives already pivoting to requiring any numerical figures to be in codebases that are regularly reviewed by AI.

However, (2) is where I think the biggest long-run value is going to come from. Take the comparison of value-of-skill to verifiers above. Do you have clean code? Can you intuit basic statistical arguments? With AI, we now have an infinite number of verifiers for the vast majority of these fundamentals. Beyond some level, I still think we have increased the number of verifiers via human+AI pair interviews, and I suspect this whole (dotted) line shifts right a good bit over the next few years.

New equilibrium prediction

I’ve heard a lot of discussion among friends and colleagues in the industry about the rampant use of AI by candidates in interviews. This is bad but only really a short-term issue. First of all, there are probably some cases where we want people to use AI for coding in interviews, because it is a tool they will use heavily on the job. Second, when we don’t want candidates to use AI, there will eventually be better automated screening tools that I suspect will make it easier to detect and limit non-approved AI use. I don’t know what those will look like, but it’s fixable with enough monitoring or proof of work.

What will be much more persistent is the use of AI to gauge skills. Imagine you’re looking for a job and believe yourself to be in the top 1% at some skill. Today, it can be difficult to get the attention of recruiters/hiring managers, and maybe equally difficult to get past flawed screens (which, as noted, may be run by people not in the top 1% of that skill). There has always been a lot of money in solving this problem (both by candidates and employers), but verification is a time-consuming process and until AI there was no real substitute for human evaluator time.

Now imagine, instead, that a company offered a credentialing service: talk to our AI as long as you’d like, as frequently as you’d like, for a small fee, and we’ll have it return a shareable+verifiable summary of your skill level, strengths, and weaknesses. Then, to firms, the company sells the highest ranked candidates as leads for relevant jobs or posts the scores publicly with verifiable aliases. The company I envision doesn’t exactly exist yet (at scale) and, if I were a start-up kind of guy, this is the type of company I would be trying to build.

I’m far from the first to think using AI for interviews in one way or another (e.g., 1, 2) – so maybe I’m just saying I’m very bullish on those solutions in the long run. In the short run, I suspect the main challenges to growing this market are that it’s hard to prove the quality of any ranking/scoring approach (after all, that’s the reason for the labor market mismatches in the first place), and that there really needs to be either a small number of platforms or a relatively small number of evaluation standards/skills. “Small” can still be a handful though (we don’t need a single platform to dominate), becuase again if you are a top candidate it’s easy to spend a few days or a week passing public evals to get past screens.

I’ll make the following few additional quick predictions that hint at path toward a new equilibrium (ranked from most to least confident)

1. Technical salary distributions will widen, as screening improves and wages better approximate skill

2. Drawn in by new top salaries, entry into technical fields will increase (and I suspect academia and government will struggle more to find and keep good talent)

3. As a result of this expanded entry (depending elasticity of supply/demand and up-skilling from AI), either or both of

   • Lower average/median DS compensation (though again, higher for the top of the distribution), or
   • More DS positions in general, because the ROI on low-skill DS have increased via reduced compensation

I expect that AI tools for augmenting on-the-job DS work will counfound some of this, so think of these predictions as layering on top of that dimension of AI impact.