My greatest frustration with using modern AI tooling (i.e., claude, chatGPT, etc.) stems from the fact that I can’t easily run these models on my humble local computer. I have to rely on an HTTP API and sign-up for all the fun downtime involved. If any combination of the public internet, my outbound network infrastructure or (likely) the AI firm’s compute overwhelmed by other user traffic, I get a network communication error response in place of a generated AI response by this allegedly transformational technology.
The size and scope of AI models has ballooned to an almost comedic scope. You have models regularly with a trillion parameters requiring GPU clusters to run. And businesses are building data centers specifically to run these large models.
The latency is a bit annoying too.
Comparing the first conversational ChatGPT 3.5 models that generated heavy buzz back in 2022 to the behemoth models running today is stark in both the depth and quality of the responses, but also in the response time. Part of the increased latency comes from changes in model architecture (the advent of “reasoning” models), but also in the size of the models as well (there’s a lot more math that goes into output tokens in current models).
It feels reminiscent of the so-called “lazy boys” era (1980s-1990s approximately) except instead of waiting twenty minutes for a compiler to spit out some linking error, now you wait five to ten minutes for thousands of lines of code you’ll probably not spend much time reading prior to issuing another command to your AI servant.
In some regard, I suspect some of the capabilities of large models boil down to acting as a bonafide compressed database of the internet. I didn’t do the exact math on it, but google tells me ChatGPT 3.5, a small model, runs a footprint of about 350 GB in terms of GPU RAM. By contrast, the entirety of English wikipedia, including media assets, only runs up to 160 GB! In short, ChatGPT 3.5 was pretty big, one might even say large. And it pales in comparison to the memory footprint of current state-of-the-art models (Astra 6 is what? Terabytes in size?).
As far as I can tell, the trend here won’t change anytime soon. Models will continue to grow in size, so long as that growth yields some performance gain, and I’ll be stuck spamming the “retry” button until enough compute gets built out, or I figure out why my ISP hates me in particular.