swyx calls for larger 'vintage' chat model trained on pre-AI-slop data
swyx flags a gap in the model landscape: chat-tuned vintage models don't exceed 4B parameters, leaving a practical use case unserved.
swyx posted on X calling for a chat-tuned 13B "vintage" model — one trained on internet text predating the surge of AI-generated content. In the thread, he notes that existing vintage models top out at under 4B parameters, too small for many practical applications.
The appeal of vintage models centers on their training data. Proponents use the term "low-background tokens" — borrowed from the concept of low-background steel, metal cast before atomic testing introduced ambient radiation — to describe corpora drawn from the pre-LLM-saturation web. For practitioners building evals, fine-tuning datasets, or retrieval systems, a larger vintage model could offer responses shaped by human-written text rather than synthetic content that may carry its own model-induced biases. The segment remains a visible gap: capable enough to use in production chat contexts, but currently unoccupied at the 13B scale swyx is pointing to.