Abstract

Streaming visual geometry transformers enable online 4D perception, but long-horizon deployment exposes a growing key-value cache bottleneck. Prior compression methods reduce this cost through token merging or temporal eviction, yet they usually operate on tokens instead of physical entities, which can mix geometry across objects or discard useful dynamic history. Zip-VGGT addresses this gap with object-aware spatiotemporal KV control: it uses external entity regions for object-bounded spatial merging, motion-aware temporal budget allocation, static and dynamic object-bank retention, and task-aware history access, while keeping the pretrained backbone fixed. Experiments on depth, pose, and reconstruction benchmarks show that Zip-VGGT keeps memory bounded, remains executable on long streams, and preserves competitive geometric and pose quality.

Motivation

• Unbounded Historical KV Cache: explicit historical KV caches accumulate continuously as the stream grows, increasing memory usage and attention cost.
• Boundary-Agnostic Spatial Compression: generic token merging can mix features from different physical entities and blur fine 3D geometry.
• Motion-Agnostic Temporal Eviction: token-, frame-, or attention-score-level retention remains weakly coupled to object identity and motion.

Zip-VGGT Framework

Zip-VGGT treats the historical KV cache of a streaming geometry transformer as a fixed-budget memory rather than an ever-growing prefix. Each incoming frame is paired with entity regions that are associated over time and projected to the token grid. The controller merges KV states within object boundaries, reserves shared anchors and a short recent window, allocates the remaining budget with motion-aware object scores, and stores selected entries in static and dynamic object banks.

• Object-Bounded Spatial KV Merging: Entity regions restrict KV aggregation within physical object boundaries, reducing spatial redundancy while avoiding cross-object feature mixing.
• Motion-Aware Temporal Allocation: Lightweight utility-cost scores allocate cache budgets to histories that are large, recent, query-relevant, or dynamic, while anchor protection and a diversity reserve keep useful context available.
• Object-Level Retention and Access: Static and dynamic object banks retain selected memory entries, and task-aware history access suppresses stale dynamic keys for pose-sensitive queries while leaving geometry queries unchanged.

Qualitative Comparisons

Qualitative Comparison 1

Qualitative Comparison 2

Conclusion and Limitations

Zip-VGGT addresses long-horizon streaming visual geometry by replacing an ever-growing historical KV cache with object-aware fixed-budget memory. Its core idea is to use physical entities as the shared unit for object-bounded spatial merging and motion-aware temporal allocation, so memory control remains coupled to geometric boundaries and dynamic relevance without modifying the streaming backbone.

The method is scoped as a bounded-memory controller for streaming geometry. Because memory organization uses external object regions, fragmentation, ID switches, occlusion, tiny fast objects, and crowded scenes can affect grouping and allocation quality. Future work should amortize object-prior generation and model object uncertainty inside the controller.