“you absolutely have to view LLM benchmarks from a position of default-distrust” @ouguoc.mastodon.online.ap.brid.gy describes how easily answers to benchmark problems can leak into the training set. ‪https://seinmastudios.com/posts/llm-benchmarks-are-not-trustworthy/

why is the link not a link? let's try that again: seinmastudios.com/posts/llm-be...

Your piece is great. As usual. Yeah, I think there's no getting around that we can't measure human welfare without a set of values to define what that means. There's not some "scientific" trick or shortcut that lets us authoritatively, universally, say that this is better than that. 1/

Technocrats pretended that GDP was that for a while, and it kind of worked because, broad brush, the correlation was pretty strong between GDP per capita and qualitative, intuitive, perceptions of prosperity and satisfaction. 2/

It was a great fit for "neoliberal" economics, whose ideological trick was largely to obscure methodological problems economists had long discussed and pretend something like economics 101 provided a scientific basis for policy to which intelligent people must defer. 3/

Besides the helpful casual empirics, it had this story, for market economies, GDP is the quantity of the highest value (because market optimized) basket of goods and services produced by the economy, and so that quantity, a simple number, should be a pretty good proxy for wealth. 4/

But real-life markets are imperfect optimizers, and even theoretical markets are arguably local rather than global optimizers that might get stuck in bad path dependencies (like automobile-dependent low density living, I'd argue). 5/

This trick of replacing an infinitely dimensioned "what" with a single number "how much" just doesn't work. 6/

So we are left with judgment calls to make, which invariably involve both evaluating tradeoffs on dimensions we'd mostly all agree are valuable (say shipbuilding capacity vs health care services), but also require imposing contested values. 7/

Some people think the costs of low-density single-family infrastructure are totally worth it, resources spent in support of human flourishing. I think an auto-centric built environment is both costly and inferior in welfare terms to achievable alternatives. 8/

Each claim requires assertions about other people's preferences in a normative sense, what they should want (since it's indeterminate what they do want while the hypothetical choice, two very different social equilibria, are not remotely before them). 9/

There's no "scientifically" right or wrong answer other ppl must defer to. Only judgements—not pulled from nowhere, informed by marshaling evidence—but judgments nonetheless, of which we have to persuade our fellow citizens, rather than truths we can discover and propound as incontrovertible. /fin

“Positive-sum solutions to multi-period stag hunts select for time consistency more than cost effectiveness.” @akhilrao.bsky.social akhilrao.org/blog/2025/07... remarkable insights what leadership and coordination actually entail, in the dry language of game theory. except with death cults. 1/

project designs demanding “unnecessarily” expensive commitments, work to ensure partners highly committed to shared values, can augur success in ways that seem irrational, inefficient, dumb to reviewers who imagine a dictator with a Gantt chart can just get every participant to play their part. /fin

i guess my lean towards (a) comes from a kind of qualitative empiricism, at best. year by year growth numbers aren’t reliable, but China’s share and increasing dominance in important sectors doesn’t require a well calibrated horserace to observe. 1/

that doesn’t mean China’s successes render its model superior in welfare terms! (there were lots of things the Soviet Union ill-advisedly produced a lot of, “dominated”). 2/

traditional growth measures are intended in a neoliberal context to obviate the question of “are we producing the right things?”, because markets optimize, then adding dollars spent on new production makes scalars, the value of “best uses” that we can rank. 3/

but i think rentierism in the US, persistent high mark-ups embedded in those numbers we sum, have weakened the historical correlation between GDP and welfare. 4/

so i think on both sides, our numbers aren’t what we want. they are overtly massaged on one side. they are undermined by structural change despite consistent, earnest, econometrics on the other. 5/

so ultimately we have little choice but to rely on course, qualitative observations, and impose our own weightings on them. 6/

does the military heft and/or generalized physical-world capacity that comes with shipbuilding overwhelm the cost in all the services thousands of workers in shipbuilding might otherwise supply? 7/

i have a view, but it could be wrong! still, i don’t think there is anything we can straightforwardly measure to decide the question, even if the measures weren’t massaged. /fin