"My experience, at least for now, is that it can complement people who know how to use it, but risks shortcutting those earlier in their careers before they’ve learned the building blocks."
That's my big concern.
How much of your ability to formulate useful questions and suss out bullshit in the results is from an earlier period of having mucked around in the data and developed reality-based intuition about it?
What's the path for a current 18-year-old?
When the machine can relatively easily get them a thing that looks like an answer, when do they develop the internal model of the world that enables them to usefully poke at the results?
I honestly don’t know. I tried to highlight how my experience helps me execute this well (realizing I didn’t mention I was a load-and-stress tester at one software point, and have a good instinct on where to check results in the process). Maybe education focuses much more on validation and checking, but it’s a tough thing to gauge.
You are very much correct in your assessment. I am a software engineer, 62 years old now, but still learning new stuff every day. And yes, I use AI as well in my work. The pace at which AI can code I can never reach, and I don’t have to. But, and that’s a serious but, you have to be sharp on what it delivers. You need to know what you (but very much so “it”) is doing.
My output is certainly higher than before, but my work changed. I am more of a development manager and software architect now, as I need to layout the groundrules for AI to adhere to. And as said, you need to stay sharp, as not always it does the correct things. And even when you notify it that it “forgot” about the rules, it will do it again, and again.
So we still have a long way to go, but as long as we keep an eye on what is going on it seems okay to me. But I hope it doesn’t take over because WE think it can do that…
Excellent post, informative and down-to-earth. Hard to find that sometimes amidst the sea of hypesters, grifters, doomers and head-in-the-sand skeptics.
Thank you. Yeah since the debate is, for good reasons, polarized between AI as slop and AI as superhuman god, I thought just demonstrating the normal ways it helps me out might advance the conversation.
I'm in a completely different field, but these use cases and your personal experience are similar to mine. I am able to do far more, far faster, and even learn more. However, the AI shifts the bottleneck from implementation to design and evaluation. With the AI implementing quickly, the pace of major cognitive decisions goes up, which is both exciting and exhausting.
Yes. I’m glad to hear the benefits are across fields, and I do have to be more conscious of what I want to explore at any moment, because i can actually make progress on it! It’s a good kind of exhausting to have.
Excellent post. While the terminal form-factors have gotten the most attention, curious if you've tried simply installing Anthropic's official Claude Code extension in the Positron IDE?
This discussion was pretty easy for me to follow, being that I have no formal training in econ, programming, or business, but I have picked up enough knowledge from being around folks who deal with computers professionally that I did get a positive benefit from reading it. Congrats on this rather unusual accomplishment! I do appreciate what you have to say. Thanks!
This looks impressive at first, but strikes me as entirely too credulous. You expended few words on how you were validating the outputs, which is where the rubber actually hits the road. For example, I wouldn't trust the tabular collations as far as I could throw them, since every model I've tried has hallucinated junk into my numerical data. And if I have to go through them with a magnifying glass, the labour savings are lost. If I must treat my tools like a student, I'd rather take the student, since at least then somebody else is benefitting.
There is a big difference between appearing productive and actually being so. Economists wonder why they suffer from physics envy. This lack of rigour is part of the reason.
"I have no idea if this analysis was executed correctly ... doesn’t disprove my instinct."
Congrats on the new confirmation bias machine! ;)
"But they’ve made exploring what’s right a lot cheaper."
Have they? Because I'm reliably told that this technology cost enormous amounts of money to create, costs enormous amounts of money to operate, does enormous damage to the environment, and does enormous damage to the political economy. Plus a not-negligible subscription fee.
So I'd say the costs are pretty high. That's provable. What's less provable is whether more or less "exploring what's right" is being done now vs previously.
My concern is that a billion people producing a thousand junk regressions a minute, which are themselves expensive summaries of previous junk regressions, will flood the analytical zone with so much slop shit that quantitative reasoning is lost completely in the garden of forking paths. Things were already really bad, now we're going to turn the quality-control knob down even further and validate with vibes?
I look at the "analysis" Claude did above and I see "Brad Pitt" fighting "Tom Cruise": its surface-level plausibility -- but complete inability to produce anything even remotely distinctive -- makes it less trustworthy, not more. Esp when there's not a payoff beyond "sometime I should look at this more" (unlikely to get to it, tho!).
Meanwhile, if a journalist wants to talk to you on demand, they can just ask Claude to create a Koncbot app overnight, they'll get close enough that it won't immediately disprove their instinct about what you might say. That way they can focus on more interesting things, like partying: https://www.astralcodexten.com/p/sota-on-bay-area-house-party
AI will devastate entry level jobs in finance. I used to see people spreading statements, then churning over writeups and then rewriting the final product after three days. The output and grammar was still mediocre. AI can do a detailed credit analysis in minutes. It can do forecasts without circular references that takes hours to figure out.
The last chart from the AIDS model doesn't make sense. By construction, Hicksian own price elasticities have to be negative. So you definitely want to double check your work before relying on AI.
you may be able to do a programmatic analysis of “interesting” by looking at a Shannon information (entropy) measure of the particular data streams and then examining the importance of that data to various demographic categories, e.g. if the price/quantity of x is unexpected, and the demographic driving tends to be a predictor…
Probably the only difference between this and what you’re doing is to use an information theoretic measure as a driver rather than have the LLM pick some criteria for interesting
Very interesting article. But I'm a little concerned by one aspect of one of your queries. You asked '...make graphics...and create a ... file ... that confirms. This seems like it's driving a conclusion rather than an analysis (confirmation bias?). You do concluded by saying 'I still believe the tools can’t identify what’s interesting or draw the right conclusions on their own'. But as usual, any analysis tool will only be as good as the data (or query) it's using or trained on and we've seen how LLM can be inherently biased. Same old adage, garbage in, garbage out, but now it's likely to create fake news or bot chatter to everyone's detriment.
Thank you. I worked in a hospital doing medical coding. The hospital is now in its second year of testing AI to do the coding and were using 'the coders on staff to audit AI's work and train AI.
This article is fascinating my big takeaway was this: "You will, no doubt, have more normal and productive projects you want to explore and see if they are worth developing fully. The real work is finding good questions and understanding how to make the results rigorous. Using the terminal gets you to where you can focus on what matters the most" I guess until AI can figure out good questions,( by capturing yours) audit and regulate itself you have time to stay put. Paul Krugman recommended this article in his Substack today. 2.26.2026
Ask one of the top paid versions (or more than one) to catalog all of Trump's lies or falsehoods since he ran for office in 2015. Categorize these by severity and the likelihood that he knew they were false, also categorize by issue area. Produce important charts. Then do the same for Biden and Obama, and compare the three Presidents across all dimensions. Include other details.
This project, done really thoroughly and well, would take a large team of skilled humans months and might cost millions. An AI could do it in a day, pretty much for free, generating an extensive report hundreds of pages long, plus a vast index of each falsehood or lie described in detail. And the results could be stunning and very important.
"My experience, at least for now, is that it can complement people who know how to use it, but risks shortcutting those earlier in their careers before they’ve learned the building blocks."
That's my big concern.
How much of your ability to formulate useful questions and suss out bullshit in the results is from an earlier period of having mucked around in the data and developed reality-based intuition about it?
What's the path for a current 18-year-old?
When the machine can relatively easily get them a thing that looks like an answer, when do they develop the internal model of the world that enables them to usefully poke at the results?
I honestly don’t know. I tried to highlight how my experience helps me execute this well (realizing I didn’t mention I was a load-and-stress tester at one software point, and have a good instinct on where to check results in the process). Maybe education focuses much more on validation and checking, but it’s a tough thing to gauge.
You are very much correct in your assessment. I am a software engineer, 62 years old now, but still learning new stuff every day. And yes, I use AI as well in my work. The pace at which AI can code I can never reach, and I don’t have to. But, and that’s a serious but, you have to be sharp on what it delivers. You need to know what you (but very much so “it”) is doing.
My output is certainly higher than before, but my work changed. I am more of a development manager and software architect now, as I need to layout the groundrules for AI to adhere to. And as said, you need to stay sharp, as not always it does the correct things. And even when you notify it that it “forgot” about the rules, it will do it again, and again.
So we still have a long way to go, but as long as we keep an eye on what is going on it seems okay to me. But I hope it doesn’t take over because WE think it can do that…
Excellent post, informative and down-to-earth. Hard to find that sometimes amidst the sea of hypesters, grifters, doomers and head-in-the-sand skeptics.
Thank you. Yeah since the debate is, for good reasons, polarized between AI as slop and AI as superhuman god, I thought just demonstrating the normal ways it helps me out might advance the conversation.
I'm in a completely different field, but these use cases and your personal experience are similar to mine. I am able to do far more, far faster, and even learn more. However, the AI shifts the bottleneck from implementation to design and evaluation. With the AI implementing quickly, the pace of major cognitive decisions goes up, which is both exciting and exhausting.
Yes. I’m glad to hear the benefits are across fields, and I do have to be more conscious of what I want to explore at any moment, because i can actually make progress on it! It’s a good kind of exhausting to have.
Excellent post. While the terminal form-factors have gotten the most attention, curious if you've tried simply installing Anthropic's official Claude Code extension in the Positron IDE?
I need to do that! Honestly I just jumped that step when I moved to Claude and went straight into the command line. But I should.
This discussion was pretty easy for me to follow, being that I have no formal training in econ, programming, or business, but I have picked up enough knowledge from being around folks who deal with computers professionally that I did get a positive benefit from reading it. Congrats on this rather unusual accomplishment! I do appreciate what you have to say. Thanks!
Really good work here. Thanks.
Great post, Mike!
Thank you!
This looks impressive at first, but strikes me as entirely too credulous. You expended few words on how you were validating the outputs, which is where the rubber actually hits the road. For example, I wouldn't trust the tabular collations as far as I could throw them, since every model I've tried has hallucinated junk into my numerical data. And if I have to go through them with a magnifying glass, the labour savings are lost. If I must treat my tools like a student, I'd rather take the student, since at least then somebody else is benefitting.
There is a big difference between appearing productive and actually being so. Economists wonder why they suffer from physics envy. This lack of rigour is part of the reason.
"I have no idea if this analysis was executed correctly ... doesn’t disprove my instinct."
Congrats on the new confirmation bias machine! ;)
"But they’ve made exploring what’s right a lot cheaper."
Have they? Because I'm reliably told that this technology cost enormous amounts of money to create, costs enormous amounts of money to operate, does enormous damage to the environment, and does enormous damage to the political economy. Plus a not-negligible subscription fee.
So I'd say the costs are pretty high. That's provable. What's less provable is whether more or less "exploring what's right" is being done now vs previously.
My concern is that a billion people producing a thousand junk regressions a minute, which are themselves expensive summaries of previous junk regressions, will flood the analytical zone with so much slop shit that quantitative reasoning is lost completely in the garden of forking paths. Things were already really bad, now we're going to turn the quality-control knob down even further and validate with vibes?
I look at the "analysis" Claude did above and I see "Brad Pitt" fighting "Tom Cruise": its surface-level plausibility -- but complete inability to produce anything even remotely distinctive -- makes it less trustworthy, not more. Esp when there's not a payoff beyond "sometime I should look at this more" (unlikely to get to it, tho!).
Meanwhile, if a journalist wants to talk to you on demand, they can just ask Claude to create a Koncbot app overnight, they'll get close enough that it won't immediately disprove their instinct about what you might say. That way they can focus on more interesting things, like partying: https://www.astralcodexten.com/p/sota-on-bay-area-house-party
AI will devastate entry level jobs in finance. I used to see people spreading statements, then churning over writeups and then rewriting the final product after three days. The output and grammar was still mediocre. AI can do a detailed credit analysis in minutes. It can do forecasts without circular references that takes hours to figure out.
One quibble: the MWG chart is beautiful.
The last chart from the AIDS model doesn't make sense. By construction, Hicksian own price elasticities have to be negative. So you definitely want to double check your work before relying on AI.
you may be able to do a programmatic analysis of “interesting” by looking at a Shannon information (entropy) measure of the particular data streams and then examining the importance of that data to various demographic categories, e.g. if the price/quantity of x is unexpected, and the demographic driving tends to be a predictor…
Probably the only difference between this and what you’re doing is to use an information theoretic measure as a driver rather than have the LLM pick some criteria for interesting
Very interesting article. But I'm a little concerned by one aspect of one of your queries. You asked '...make graphics...and create a ... file ... that confirms. This seems like it's driving a conclusion rather than an analysis (confirmation bias?). You do concluded by saying 'I still believe the tools can’t identify what’s interesting or draw the right conclusions on their own'. But as usual, any analysis tool will only be as good as the data (or query) it's using or trained on and we've seen how LLM can be inherently biased. Same old adage, garbage in, garbage out, but now it's likely to create fake news or bot chatter to everyone's detriment.
Thank you. I worked in a hospital doing medical coding. The hospital is now in its second year of testing AI to do the coding and were using 'the coders on staff to audit AI's work and train AI.
This article is fascinating my big takeaway was this: "You will, no doubt, have more normal and productive projects you want to explore and see if they are worth developing fully. The real work is finding good questions and understanding how to make the results rigorous. Using the terminal gets you to where you can focus on what matters the most" I guess until AI can figure out good questions,( by capturing yours) audit and regulate itself you have time to stay put. Paul Krugman recommended this article in his Substack today. 2.26.2026
Mike,
Here is a good AI project to try:
Ask one of the top paid versions (or more than one) to catalog all of Trump's lies or falsehoods since he ran for office in 2015. Categorize these by severity and the likelihood that he knew they were false, also categorize by issue area. Produce important charts. Then do the same for Biden and Obama, and compare the three Presidents across all dimensions. Include other details.
This project, done really thoroughly and well, would take a large team of skilled humans months and might cost millions. An AI could do it in a day, pretty much for free, generating an extensive report hundreds of pages long, plus a vast index of each falsehood or lie described in detail. And the results could be stunning and very important.