Modal Clusters are generally available
Modal 宣布 Modal Clusters 正式可用，通过一个装饰器 @modal.clustered 即可获得多节点集群，节点间经 InfiniBand verbs 通信可达 6.4 Tbps，自动配置 PyTorch 和 NCCL。
Deployment & Ops News
Latest reporting, research and product updates filed under Deployment & Ops.
Latest stories
7 stories
Confidence Thresholds for Model Escalation Routing
Sending every request to your strongest model gets good answers at the highest price, because you pay frontier rates for requests a cheaper model would have answered correctly. Fixed routing rules, such as picking the model by keyword or task type, cost less but need rewriting as your traffic changes. Confidence-based escalation sits between the two. You ask the model to score its own answer, then route on that score. Answers with a high score stay on a cheap model. Answers with a low score go t
VM Sandboxes: Full computers for agents
Today we’re making VM Sandboxes generally available on Modal, built for those who need to give their agents the power of a full computer.With one flag, you’ll get a fully capable Linux VM with all the niceties that you expect from a traditional modal.Sandbox, and it Just Works™. This brings the same APIs, modal.Images, sub-second cold-starts, and CPU/memory bursting capabilities as previously, all whilst supporting the hundreds of thousands of concurrent Sandboxes that our users are accustomed t
How to Gate Pull Requests on LLM Evals in CI
Changing one line in a support agent’s system prompt can ship an agent that tells customers the refund window is 30 days when your policy says 14. Nothing in a normal CI pipeline checks what the model says, so the build passes and the first person to see the wrong answer is a customer.Gating a pull request on a fixed eval set works the same way as gating on a failing unit test. You keep test cases in the repository, run them when a prompt changes, and block the merge when too many fail.In this g
Runtime Roundup: VM Sandboxes, Multi-node clusters, and more
Modal just hosted our inaugural conference, Runtime. Here are a few of the highlights that we announced.VM SandboxesVM Sandboxes give your agent access to a full Linux computer. Agents increasingly want to live inside something that looks like a real machine: running Docker stacks, local databases and dev servers, graphical environments and mobile simulators, and even monkeying around with the Linux Kernel itself.VM Sandboxes are already in use at customers like Linear, Legora, and Snorkel power
Cost vs. Quality Tradeoff Framework for Agent Models
Picking a model for an agent by leaderboard rank pays frontier prices for tasks that a cheaper model may handle at the same accuracy. The question to answer is not which model scores highest, but which model is the cheapest one that is good enough for the task in front of you. This guide is a three-step framework for making that call. You set the quality bar the task needs, measure cost per quality point on your own examples, and pick the cheapest model that clears the bar with margin. Tl;dr Set
Gemini 4 Argon: our next era of frontier intelligence
Sep 30, 2026 | Gemini 4 Argon delivers frontier performance in complex workflows across real-world software engineering, enterprise knowledge work like legal and finance, and cybersecurity defense. In this article Today, we’re announcing our new frontier model, Gemini 4 Argon, which is rolling out to a set of trusted cyber defenders through our Fairwind Program. Built to sustain deep reasoning across complex, long-horizon workflows, Argon is fundamentally changing the way we work and build at Go
3 stories
Building a Golden Eval Dataset from Production Traffic
You update a prompt, or a provider rolls out a new checkpoint under the same model ID, and something in production regresses. The last week of user complaints looks slightly different from the week before. A public benchmark like MMLU won’t catch that. It measures general capability across academic subjects, not how the model handles your product’s traffic. A golden eval dataset closes that gap. It’s a curated collection of production inputs paired with reviewed expected outputs, versioned in Gi
AI Agent Regression Testing After a Prompt or Model Change
A ~author/family-latest alias always resolves to the newest concrete model in a family. That’s convenient in production and a problem in a regression test, because the model can change between runs without any change in your repository. Our latest model resolution docs describe the mechanism and recommend a concrete model slug when you need a fixed version for reproducibility. This guide covers the locked case set and the behavioral contract, then the model-swap case in detail. Tl;dr Regression
Introducing GPT-6.1 Sol
Near-Astra intelligence for a fifth of the price We’re introducing GPT‑6.1 Sol, an upgrade to GPT‑6 Sol that nearly matches GPT‑6 Astra’s intelligence on agentic coding, computer use, and professional work at one-fifth of Astra’s standard input and output token prices. Cached input costs just $0.10 per million tokens—95% less than standard input pricing and 50% less than GPT‑6 Sol’s cached input pricing—giving developers more room to build and run capable agents that reuse context across request
2 stories
Taking vLLM Apart: A Practical Guide to Disaggregated Serving
A single vllm serve process does three jobs that get in each other's way: processing prompts (prefill), generating tokens (decode) and a pile of CPU work around them. Disaggregated serving in vLLM separates the different stages of LLM inference. Splitting prefill from decode stops long prompts stalling everyone else's output, as long as the KV cache moves between them fast. Moving tokenization and parsing to a CPU-only frontend (/render, /derender) takes that work off your GPU nodes and leaves t
How Databricks rolls out frontier models to 12,000 employees on Day 1
Providing our employees access to frontier AI capabilities is a top priority at Databricks, and consequently, it is important to us for them to use new models instantly when they become available. At the same time, it is nontrivial to give more than 12,000 people rapid access to a new model because:Models that are marketed as frontier often aren’t. For example, Opus 5.0 was more expensive and ranked lower on both quantitative and qualitative quality scores among our engineers compared with Opus
3 stories
Automating eval design and hillclimbing with Claude
Evaluations provide a signal on how your app or skill is performing on specific tasks. But designing evaluations, and improving performance on them without fooling yourself, is hard. We've added guidance for both to the claude-api skill. With the skill, you can run /claude-api build-eval to build an evaluation inside your codebase, and run /claude-api hillclimb to improve your application against it, one change at a time, with a held-out set of examples to catch overfitting. In this article, we
Prompting Claude Opus 5.5
This guide covers the prompting patterns specific to Claude Opus 5.5. For the model's capabilities and API changes, see What's new in Claude Opus 5.5. For techniques that apply across all current Claude models, see Prompting best practices. Claude Opus 5.5 generates output tokens more than 30 percent faster than Claude Opus 5 and tends to finish the same task with fewer tokens. Existing Claude Opus 5 prompts should perform well without changes, and the patterns in Prompting Claude Opus 5 remain
How GPU Prices Can Double While AI Gets Cheaper
GPU prices have doubled in the last six months, from $4.40 to $8.08 per GPU-hour. But AI prices are falling. How can that be? It is not a simple answer. Higher GPU costs are likely to remain while the industry races to build out new data centers. Every component of the buildout is increasing in cost, from concrete to copper to credit.1 Above all, electricity remains the limiting factor, which Oracle is experiencing : last week it invoked force majeure on its New Mexico campus after the natural-g
1 story
1 story
2 stories
3 stories
LiteParse Updates: September 2026
LlamaIndex 发布 LiteParse 2.14.6 更新，通过对自维护 PDFium fork 做内存分配优化（内置 mimalloc）等手段，将文本提取耗时降低 20-25%，平均 2.76ms/页，markdown 渲染 3.94ms/页。
Announcing vllm-metal: Concurrent Serving on Apple Silicon
vLLM 官方发布 vllm-metal v0.28.0，把 vLLM 的 V1 调度器、paged KV cache 和 OpenAI 兼容服务器带到 Apple Silicon，由 MLX 和 Metal 执行模型，版本号与上游 vLLM 对齐。
Transformers now runs llama.cpp quants
Hugging Face 宣布 transformers 支持直接运行 GGUF 量化模型，通过 from_pretrained 传入 gguf_file 即可加载 Hub 上的 GGUF checkpoint，并复用 ggml 的 Metal 内核。
1 story
2 stories
You can now train and run 500+ models locally with our Unsloth Docker image! 🐳 Use our new GUI or ...
Unsloth 宣布可使用其 Docker 镜像本地训练和运行 500+ 模型，提供新 GUI 和 notebooks 工作流，无需配置，支持 NVIDIA 和 AMD，指南见 https://unsloth.ai/docs/get-started/install/docker。
Migrating the GitHub Copilot runtime to Rust, using Copilot
GitHub 工程师 Stephen Toub 复盘用 Copilot 智能体在约 14.5 周内将 Copilot agent runtime 从 TypeScript/Node.js 全量重写为 832,378 行生产 Rust，AI 智能体完成大部分代码，共 128 个 PR 增量合入 main 并持续发布。
1 story
1 story
2 stories
DeepSeek-V4.1-Flash: more efficient prefill for coding agents
DeepSeek 本周在 HuggingFace 开源 DeepSeek-V4.1-Flash 权重，模型总参数 552B，prefill 激活 8B、decode 激活 16B，支持 1M token 上下文和文本加图像输入，已上线 Baseten Model APIs。
AI researchers debate how close we are to recursive self-improvement
Dwarkesh Patel 与 Zyphra CTO Beren Millidge、Thinking Machines 首席科学家 John Schulman、Baseten 模型训练负责人 Charlie O'Neill 三位研究者对谈递归自我改进（RSI）的前景。
1 story
3 stories
DeepSeek AI Released DeepSeek-V4.1-Flash with 1M Context, FP4 KV Cache, and Cross-Layer Attention Reuse
DeepSeek AI 发布多模态 MoE 模型 DeepSeek-V4.1-Flash，552B 主干加 196B Engram 参数，上下文窗口 1M，prefill 激活 8B 参数、decode 激活 16B，全局 KV 缓存降至每 token 890 字节，约为 DeepSeek-V4-Flash 的 1/4、DeepSeek-V1 的 1/437。
Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL
Hugging Face 在 TRL v1.14 的 AsyncGRPOTrainer 中支持只训练和同步 LoRA adapter，配合 Storage Bucket 挂载与代理路由，让训练 Job 和 vLLM 推理 Job 在不同机器上运行而无需 NCCL。
Rebuilding AUTOMATIC1111 with Gradio Workflow
Hugging Face 发布 Workflow1111，用 gr.Workflow 以 73 个节点、11 条媒体管线重建 AUTOMATIC1111 的大部分功能，覆盖文本生成图像、高清修复、图生图、prompt matrix、VLM 反推提示词、检测生成 inpaint 蒙版、ControlNet 式预处理器、背景移除、PNG Info 和图生视频。
1 story
4 stories
Latency Scaling Differences for GPT and Claude Models
Epoch AI 测量四款模型的首 token 延迟（TTFT）随上下文长度的缩放，发现 GPT-5.6 Terra 和 Sol 呈明显二次曲线，Claude Sonnet 5 接近线性，Opus 5 噪声较大但仍接近线性。
vLLM x AgentX: Optimizing for Real-World Agentic Serving
vLLM 团队发布针对智能体负载优化的博客，基于 SemiAnalysis 的公开 AgentX 基准展示成果：DeepSeek V4 Pro 在 GB300 上达到 83K total tokens/GPU-秒。
Give any model a terminal and files
OpenRouter 发布 openrouter:shell 服务端工具和 Files API，OpenRouter 上任何支持工具调用的模型都可在托管 Linux 容器中运行命令，两者现已在 beta 阶段开放。
Inside the megakernel serving engine for North Mini Code
Cohere 发布为 North Mini Code 构建的围绕 decode megakernel 的推理引擎，BF16 下单张 H100 端到端解码吞吐比 vLLM 快 1.25–1.41 倍，batch size 1 时达 292 tok/s（SoL 的 62%），代码已在 GitHub 开放。
2 stories
Excited to see early customers already using Astra on Azure! https://azure.microsoft.com/en-us/blog/...
Satya Nadella 发文表示，早期客户已开始使用 Azure 上的 Astra。GPT-6 Astra 现已通过 Microsoft Foundry 提供，详情见 Azure 官方博客 https://azure.microsoft.com/en-us/blog/gpt-6-astra-frontier-intelligence-for-work-now-available-in-microsoft-foundry/。
Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson
NVIDIA 发布教程，讲解在 Jetson 上部署新一代开放推理模型的方法，以 Nemotron 3.5 Lightning 和 Qwen3.8-27B 为例。
1 story
2 stories
Qwen3.8-Flash can now run 1.7× faster locally with MTP!⚡️ GGUFs can reach 170 tokens/s on a RTX PRO...
Unsloth 通过 MTP（Multi-Token Prediction）让 Qwen3.8-Flash-Next 本地推理提速约 1.3 至 1.7 倍且精度不变，GGUF 在单张 RTX PRO 6000 上可达 170 tokens/s（基线 100 tokens/s）。
LongCat-2.0 is now free to try in @cline ! 🐱
LongCat-2.0 现已在 @cline 中免费试用！🐱 [引用 @cline]：LongCat-2.0 目前在 Cline 中免费使用。 这是一款来自 @Meituan_LongCat 的 1.6T 开源权重 MoE 模型，拥有 1M 上下文，得分与 Claude Opus 4.7 和 Gemini 3.1 Pro 相近。 立即尝试：npm i -g cline /model 在免费模型下选择 LongCat-2.0
1 story
1 story
1 story
1 story
2 stories
GLM-5.3-Flash can now be run locally! ✨ Run 3-bit on 128GB RAM via Unsloth GGUF. GLM-5.3-Flash (ox...
Unsloth 发布 GLM-5.3-Flash（ox-alpha）的 GGUF 量化版本，可在 128GB RAM 设备上以 3-bit 运行。该模型为 Z.ai 的 320B-A18B 开源多模态模型，MIT License 发布，原文称其在 DeepSWE、编码和智能体基准上媲美 Claude Opus 4.8。
Blog MiniMax-H3 on 8×H200: 1.95× Lossless, Up to 6.24× at 0.76–0.91 SSIM We benchmarked MiniMax-H3 video generation on 8× NVIDIA H200 with SGLang Diffusion, holding prompts, seeds, resolution, frame rate, and denoising steps fixed across six workloads. - SGLang's dense, l... SGLang Diffusion Team, Cache-DiT Team, NVIDIA, Ant Group
SGLang Diffusion 团队在 8×NVIDIA H200 上对 MiniMax-H3 视频生成进行基准测试，其密集无损路径较 Diffusers 快 1.85–1.95×，无近似损失。
2 stories
VMs won't contain cyber-capable agents
Trail of Bits 在 Patch the Planet 项目中获得 GPT 5.6-Cyber 预览权限，让其挑战逃逸 Debian 12 主机上的 QEMU/KVM 虚拟机，结果以三种不同方式成功逃逸。
DeepSeek V4 Pro: Tops SWE-Bench & Cuts Cost per Task by 3x vs. Fable 5
Fireworks 上线 DeepSeek V4 Pro 0813，提供 serverless、专属部署及 SFT/DPO/RFT 训练支持，模型具备 1M token 上下文窗口和原生工具调用。
2 stories
Dylan Patel – Anthropic & OpenAI will have most of the world’s compute by 2028
在最新一期播客中，SemiAnalysis 创始人 Dylan Patel 与 Dwarkesh Patel 讨论实验室经济学，预计 Anthropic 和 OpenAI 到 2028 年将控制全球大部分可用 FLOPs，因其能更好变现算力并出价高于其他方。
OpenRouter Video Generation API: A Code-First Guide
OpenRouter 推出统一的异步视频生成 API，通过 POST /api/v1/videos 提交任务、轮询状态并下载 MP4，支持 Seedance、Veo、Wan 等模型，切换模型只需更改 model 标识符。
1 story
1 story
Showing the latest 55 of 555 stories.