SpecMD: 关于推测性专家预取的综合研究
研究团队开发了SpecMD，这是一个用于在各种硬件配置上对临时缓存策略进行基准测试的标准化框架。该研究聚焦于混合专家模型，这类模型虽然实现了稀疏专家激活，但需要专家缓存机制才能将稀疏性转化为实际性能提升。此前的研究提出了以硬件为中心的缓存策略，但不同缓存策略之间以及它们与不同硬件规格之间的相互作用尚不明确。SpecMD框架旨在填补这一理解空白，系统性地评估缓存策略的交互影响与硬件适配性。
MoE推理的缓存策略一直靠经验摸，Apple给的标准化框架能系统比较不同策略，做分布式推理的可以省些心力。
AuthorsDuc Hoang, Ajay Jaiswal, Mohammad Samragh Razlighi, Minsik Cho
Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model’s parameters is used during each inference. However, to translate this sparsity into practical performance, an expert caching mechanism is required. Previous works have proposed hardware-centric caching policies, but how these various caching policies interact with each other and different hardware specification remains poorly understood. To address this gap, we develop SpecMD, a standardized framework for benchmarking ad-hoc cache policies on various hardware configurations. Using SpecMD, we perform an exhaustive benchmarking of several MoE caching strategies, reproducing and extending prior approaches in controlled settings with realistic constraints. Our experiments reveal that MoE expert access is not consistent with temporal locality assumptions (e.g LRU, LFU). Motivated by this observation, we propose Least-Stale, a novel eviction policy that exploits MoE’s predictable expert access patterns to reduce collision misses by up to 85× over LRU. With such gains, we achieve over 88% hit rates with up to 34.7% Time-to-first-token (TTFT) reduction on OLMoE at only 5% or 0.6GB of VRAM cache capacity.
Related readings and updates.
Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Key-Value (KV) cache grows linearly with the extended dialogue history, causing the model’s memory footprint to quickly exceed device limits. While recent KV cache compression methods attempt to reduce memory usage, most apply cache eviction after processing the entire…
Serving transformer language models with high throughput requires caching Key-Values (KVs) to avoid redundant computation during autoregressive generation. The memory footprint of KV caching is significant and heavily impacts serving costs. This work proposes to lessen these memory requirements. While recent work has largely addressed KV cache reduction via compression and eviction along the temporal axis, we argue that the depth dimension offers…
来源：Apple Machine Learning Research（RSS） · machinelearning.apple.com