vToken proposes token-level virtualisation for reclaimable model caches

vToken · research · 14 August listing. The preprint proposes token-level virtualisation for reclaimable key-value caches, a system-level approach to reducing inference memory pressure. It is promising for multi-tenant serving but lacks independent production validation. arXiv.