Every time a new language model ships, the first question is usually: β€œwhere is it on the leaderboard?” That question is natural, but misleading. A single rank compresses dozens of distinct capabilities into one number β€” and in that compression, exactly the thing that matters for your decision gets lost.

Models are like humans: someone who is strong at maths is not necessarily a good writer. Asking β€œwho is the smartest?” is almost meaningless; the right question is β€œfor this specific job, who is the better fit?” Instead of ranking, we should profile.

What ranking hides

A model might excel at mathematical reasoning yet slip when generating structured output. Another might follow precise instructions perfectly, yet forget details buried in the middle of long documents. If you only look at the overall rank, these variances remain invisible β€” until they surface in production as an expensive error.

The problem with the leaderboard is not that it is wrong. It is that it does not ask your question. Your question is always specific: how well does this model do my job?

Instead of a rank, a capability profile

Profiling means scoring each model across several independent dimensions rather than a single axis. In practice, the dimensions that matter are usually these:

  • Instruction following β€” when you set an explicit constraint, how precisely does it hold to it?
  • Structured-output stability β€” does it produce valid, repeatable JSON or the requested format?
  • Long-context recall β€” does it find a detail buried in the middle of a long input?
  • Reasoning depth β€” on multi-step problems, does it sustain the chain of logic to the end?
  • Refusal behaviour β€” where does it correctly say β€œI don’t know,” and where does it answer incorrectly with false confidence?
  • Latency and cost β€” does the quality it delivers justify the time and financial cost?

The key point is that none of these is general; you must measure them against your own work. Build a small evaluation set of real or near-real examples β€” even 20 to 30 cases are highly informative β€” and score each model on those exact examples, dimension by dimension.

A simple sketch for profiling

The idea fits in a few lines of code. Turn each dimension into a probe: a function that takes a model’s output and returns a score between zero and one. Then run those same probes across every candidate model and build a profile.

# Profile, not rank: score each model across independent dimensions.
# `probes` maps a dimension name to a function that evaluates a model's
# output on one example and returns a score in [0, 1].

def profile_model(run, dataset, probes):
    scores = {trait: [] for trait in probes}
    for example in dataset:
        output = run(example["prompt"])          # one call to the model
        for trait, probe in probes.items():
            scores[trait].append(probe(example, output))
    # mean of each dimension -> that model's profile
    return {trait: sum(v) / len(v) for trait, v in scores.items()}


def valid_json(example, output):
    import json
    try:
        json.loads(output)
        return 1.0
    except ValueError:
        return 0.0


probes = {
    "structured_output": valid_json,
    "instruction_following": follows_constraints,   # your own probes
    "long_context_recall": finds_buried_fact,
    # ... whatever dimension matters for your job
}

profiles = {name: profile_model(run, dataset, probes)
            for name, run in candidate_models.items()}

The output is no longer a single number; it’s a table showing each model’s strengths and weaknesses side by side. Now the choice is an informed judgment: you pick the model whose profile matches the shape of your work β€” not the one that ranks higher on some arbitrary axis.

Why this makes better decisions

It has three clear benefits. First, your choice is tied to your job, not to a generic benchmark that may have nothing to do with your need. Second, when a new model ships, you do not have to rely on rumour; you run the same probes and within minutes know whether it is better for your job or not. And third, profiling surfaces weaknesses before production β€” where fixing them is cheap, not where it is expensive.

Where profiling loses

Profiling is not free, and it is not always the right call. Building and curating the evaluation set requires real upfront effort, and that set must be maintained as the job drifts β€” a profile measured against last quarter’s cases can quietly go stale. A set of 20 to 30 examples is informative but small enough to be noisy: a single-case swing moves a dimension’s score, and tuning your choice too tightly risks overfitting to the evaluation rather than the job. For a throwaway script, a one-off task, or a low-stakes pick, standing up the whole probe harness is overkill β€” there, a leaderboard or arena rank is a perfectly good cheap heuristic. Profiling earns its cost when the decision is durable and the failure is expensive; below that line, ranking is the honest shortcut.

The leaderboard is good for a headline. A durable engineering decision needs a profile. For those, don’t rank your models; profile them.