Prior ultra-long video methods pour effort into complex hierarchical or graph memory at build time — before the query is even known. MERIT flips this: it builds a simple episodic multi-key memory for high-recall retrieval, and defers query-specific reasoning to inference time via on-demand neighbor filtering. Without building complex, high-cost memory, MERIT achieves state-of-the-art results on EgoLifeQA, LVBench, and Video-MME (Long).
0:00
2×
1.7×
1.5×
1.2×
1×
A simple yet effective agentic framework — MERIT — that prioritizes high-recall retrieval and defers semantic composition to inference time.
Motivation
Good retrieval doesn't need complex memory — simple, raw evidence is enough.
When a video stretches from hours to days, you can't feed it into an MLLM end-to-end. So the field splits the task into two stages: first build an external memory from the whole video — before any question is known — then, at inference time, retrieve from it to answer each query.
(a) Prior methods build hierarchical, graph-based memories and aggregate across them at inference.
(b)MERIT builds a simple episodic multi-key memory and, at query time, retrieves matched clips with on-demand temporal expansion.
(c) MERIT outperforms EgoRAG and WorldMM across EgoLifeQA, Video-MME (Long), and LVBench.
The problem
Prior work over-builds complex memory — hierarchies and graphs — running the strongest MLLMs over the whole video in many passes. Because it's query-agnostic, this expensive abstraction may not match what is ultimately asked.
Our take
Keep the memory simple and high-fidelity, and defer high-level relation reasoning to inference time — when the query finally tells you which relations and abstractions actually matter.
Key idea
Intelligence Allocation. Spending your strongest MLLM on preprocessing done before the task is even specified is wasteful. Preserve high-fidelity episodic evidence and defer semantic composition to query time. The bottleneck then shifts from pre-building static semantics to high-recall retrieval that reliably surfaces the right evidence.
MERIT turns this idea into two lightweight components: a simple, high-recall episodic memory, and query-aware refinement that composes the right evidence on demand at inference time — detailed below.
Method
MERIT = Multi-key Episodic Retrieval with Inference-time Temporal expansion.
Stage 1 — Episodic memory construction. Each 30-second clip is densely captioned, then a multi-key extraction derives four complementary keys per clip, forming a lightweight key–value memory M.
Stage 2 — Agentic retrieval & answering. The solver regenerates a query, matches it against the keys (key-wise max similarity) to retrieve relevant clips, and neighbor filtering expands the temporal context to distill query-relevant evidence for the final answer.
1
Multi-Key Memory
A clip can be relevant for very different reasons — a physical action, a spoken mention, an object being handled, or just coarse scene context. Collapsing all of that into a single index is brittle. Instead, MERIT attaches four complementary keys to every 30-second clip, and scores relevance by the maximum similarity across a clip's keys — so each clip can match through its best-aligned aspect. This late-interaction matching is what drives MERIT's high recall.
🎬
Event / Action
Observable physical actions and interactions between entities.
💬
Dialogue / Mention
Spoken content, exact words, or commands from the audio track.
📦
Object
Items handled, requested, or moved, along with state transitions.
📝
Summary
A compact keyword-style abstraction of the clip's core narrative.
⏱️ DAY 2 · 11:32:30
Chatting over a meal at the dining table
📄 CaptionI pick up the bowl of food and take a bite. … I reach for the cup, pick it up and take a sip, then set it down. Alice asks, “Are you going to a concert or something?” — the group chats about what's fun in Zhengzhou.I pick up the bowl of food. Lucia says, “Let me try again.” I set it down, pick it up again, and take a bite. Alice says, “Hey, hey, I'm really curious about your plan.” I look at my colleagues. Tasha says, “Ah, I'm really curious too.” Tasha adds, “I want to know too.” Alice says, “I also want to know what fun stuff is there in Zhengzhou.” I glance back at the bowl. Tasha asks, “No idea, what is there to do?” I reach for the cup, then pick it up and take a sip. Katrina says, “Check out the guides made by non-locals and you'll know.” Alice says, “Exactly, hahaha.” Tasha says, “Just got a call today saying to go play in Zhengzhou.” I set the cup down. Alice asks, “Are you going to a concert or something?” I look at my colleagues. I check my phone. Tasha says, “No idea.”
🎬
EventI pick up the bowl of food and take a bite
💬
DialogueAlice asks if I'm going to a concert or something
📦
ObjectCup is picked up and sipped from, then set down
📝
Summaryeating, sipping while discussing Zhengzhou plans
⏱️ DAY 4 · 12:57:30
Sorting delivery packages at home
📄 Caption“What's this?” I ask, then identify it: “Butterfly wind chime.” … I hand the items to Katrina, who says, “This is my materials pack… doing crafts again today.” I head outside with a garbage bag, toss it into the box, pick up another package, and read the tracking number.I take it out. “I don't know—let me check,” I say. “What's this?” I ask, then identify it: “Butterfly wind chime.” Lucia says, “It's mine.” “Okay,” I reply, handing the items to Katrina. Katrina says, “This is my materials pack… Doing crafts again today. Did makeup yesterday, don't feel like doing it today.” I head outside with a garbage bag, toss it into the box, pick up another package, and read the tracking number. I grab another, check the tracking again—still not it—so I set the rest aside.
🎬
EventI head outside and toss a garbage bag into the box
💬
DialogueKatrina says this is her materials pack — doing crafts again today
📦
ObjectI pick up packages and read tracking numbers, then set the rest aside
📝
Summarypackage sorting, check tracking, hand wind chime to Katrina, dispose garbage
⏱️ DAY 6 · 15:08:30
Charging a power bank, then heading upstairs
📄 CaptionI pick up the thread, plug the cable into the power bank, and check the battery level. Alice asks, “Are you really not inviting me to your house?” I start up the stairs, steadying myself on the railing with my left hand, and take out my phone.Alice says, “I don't even know where your door opens.” I hesitate and glance left. Others say, “It's better not to know.” I pick up the thread, plug the cable into the power bank, and check the battery level. Alice asks, “Are you really not inviting me to your house?” I turn right and walk toward the living room door. Katrina says, “If I invite you, I won't add any fun.” I glance left and start up the stairs, steadying myself on the railing with my left hand, and continue up. I take out my phone. Shure says, “Under the vase.” I swipe up and tap my phone. Katrina says, “Ah, under the vase.” I walk forward, glancing right. Others ask, “On the first floor?” Katrina says, “Maybe the girls found it.” I see Shure.
🎬
EventI start up the stairs, steadying myself on the railing with my left hand
💬
DialogueAlice asks: are you really not inviting me to your house?
📦
ObjectCable is plugged into the power bank and the battery level is checked
📝
Summaryplug power bank, check battery, walk to living room, climb stairs
2
Neighbor Filtering
The evidence needed to answer a question often spans several nearby moments, not a single 30-second clip. So when a clip is retrieved, MERIT attaches its temporally adjacent neighbors (±Δ) on demand — the simplest possible graph over the timeline — then uses the solver as a query-aware filter to distill only what's relevant. Coherent context is reconstructed at inference time, with no pre-computed hierarchies.
↔️
Temporal expansion
Pull in ±Δ adjacent clips around each retrieved anchor (e.g. ±1 min).
🔎
Query-aware refinement
The solver filters the expanded context, keeping only query-relevant evidence.
Q. While I was cleaning the egg mixer, who joined in to help?
A. Alice
B. Tasha
C. Shure
D. Lucia
Query time | DAY1 19:37:51 · Target time | DAY1 14:01:00–14:01:30
DAY1 14:00:00 ± neighbor+DAY1 14:00:30 retrieved+DAY1 14:01:00 ✓ answer here
the retrieved clip alone misses it — a ±Δ neighbor holds the answer
🔎 Filtered evidenceI hand the egg beater to Shure, who operates it to show me how — Shure joined in to help.
AnswerC. Shure
Q. While puzzling, who am I guiding to use the app on the phone?
A. Lucia, Shure
B. Katrina, Alice
C. Shure, Tasha
D. Alice, Shure
Query time | DAY1 12:46:46 · Target time | DAY1 12:31:53–12:32:03
DAY1 12:30:30 retrieved+DAY1 12:31:00 ± neighbor+DAY1 12:31:30 ✓ answer here
the retrieved clip alone misses it — a ±Δ neighbor holds the answer
🔎 Filtered evidenceI check Katrina's app on her phone, then take Alice's phone to look — I'm guiding Katrina and Alice.
AnswerB. Katrina, Alice
Q. Who joined in while me and Shure were setting up the shelf?
A. Tasha
B. Katrina
C. Lucia
D. Alice
Query time | DAY1 19:40:55 · Target time | DAY1 12:54:00–12:54:30
DAY1 12:53:30 ± neighbor+DAY1 12:54:00 retrieved+DAY1 12:54:30 ✓ answer here
the retrieved clip alone misses it — a ±Δ neighbor holds the answer
🔎 Filtered evidenceAs the shelf is set up, Katrina joins in and operates it with us — Katrina joined in too.
AnswerB. Katrina
State-of-the-Art Results
New SOTA across ultra-long and hour-long video QA.
71.2
EgoLifeQA
+5.6 vs. SOTA
71.8
LVBench
+9.9 vs. SOTA
77.7
Video-MME (Long)
+1.1 vs. SOTA
0.56
Hit rate (highest)
simplest memory
The simplest memory, yet the best retrieval.
MERIT uses only single-granularity 30-second clips with lightweight multi-keys — no hierarchies, no graphs — and still attains the highest hit rate and highest accuracy, confirming that precise retrieval is the real bottleneck in ultra-long video QA.
Results on Ultra-Long Video QA (EgoLifeQA)
Average accuracy · by backbone
Backbone
Uniform Hierarchical Graph Ours
48.6GPT-5
52.0EgoRAG
65.6WorldMM
71.2MERIT
46.4Gemini
57.5EGAgent
64.2MERIT
38.6Qwen
56.4WorldMM
54.2MERIT
Average accuracy (%) — best of each type, within the same backbone.
Per-QA-type accuracy (%) on the GPT-5 backbone (axes 0–80).
Show full EgoLifeQA table
Model
EntityLog
EventRecall
HabitInsight
RelationMap
TaskMaster
Avg
MLLMs (Uniform Sampling)
Qwen3-VL-8B
35.2
30.2
39.3
46.4
46.0
38.6
Gemini 2.5 Pro
43.2
40.5
41.0
55.2
52.4
46.4
GPT-5
47.2
42.1
47.5
53.6
55.6
48.6
Hierarchical Memory Based
EgoRAG (GPT-5)
40.0
56.3
62.3
54.4
52.4
52.0
Ego-R1 (3B)
51.2
53.2
63.9
50.4
50.8
53.0
Graph Memory Based
LightRAG (GPT-5)
40.8
48.4
67.2
50.4
44.4
48.8
HippoRAG (GPT-5)
48.8
60.3
70.5
60.8
66.7
59.6
Video-RAG (GPT-5)
49.6
56.3
67.2
55.2
54.0
55.4
HippoMM (GPT-5)
45.6
53.2
70.5
55.2
58.7
54.6
M3-Agent (7B)
44.4
54.8
62.3
56.8
54.0
53.5
EGAgent (Gemini 2.5 Pro)
54.4
57.1
60.3
62.4
74.6
57.5
WorldMM (Qwen3-VL-8B)
49.6
56.4
63.9
58.4
58.7
56.4
WorldMM (GPT-5)
62.4
64.3
75.4
62.4
71.4
65.6
Ours
MERIT (Qwen3-VL-8B)
43.2
54.0
67.2
53.6
65.1
54.2
MERIT (Gemini 2.5 Pro)
60.8
61.1
65.6
65.6
73.0
64.2
MERIT (GPT-5)
67.2
70.6
73.8
74.4
71.4
71.2
Results on Hour-Long Video QA (LVBench & Video-MME (Long))
LVBench · GPT-5 backbone
MERIT
71.8
WorldMM
61.9
Uniform
60.4
Video-MME (Long) · GPT-5 backbone
MERIT
77.7
WorldMM
76.6
Uniform
74.3
Accuracy (%), axis starts at 50 · the longer the video, the larger MERIT's lead.
Show full LVBench & Video-MME (Long) table
Model
LVBench
Video-MME (L)
MLLMs (Uniform Sampling)
Gemini 2.5 Pro
57.0
55.7
GPT-5
60.4
74.3
Hierarchical Memory Based
EgoRAG (GPT-5)
32.2
41.1
Ego-R1 (3B)
34.1
42.7
Graph Memory Based
LightRAG (GPT-5)
30.4
46.6
HippoRAG (GPT-5)
54.0
52.1
Video-RAG (GPT-5)
33.1
55.4
HippoMM (GPT-5)
38.2
41.6
M3-Agent (7B)
49.3
55.3
EGAgent (Gemini 2.5 Pro)
–
74.1
WorldMM (GPT-5)
61.9
76.6
Ours
MERIT (Gemini 2.5 Pro)
–
76.3
MERIT (GPT-5)
71.8
77.7
Memory Building Cost
Against the graph-memory SOTA (WorldMM), MERIT builds its memory once per clip instead of re-reading and consolidating across many scales.
8.4×
fewer input tokens 815K vs. 6,860K
8.6×
fewer output tokens 288K vs. 2,472K
3.8×
fewer MLLM calls 6,223 vs. 23,731
For more ablations and analysis, please refer to the paper.
Qualitative Results
On EgoLifeQA: precise multi-key matching localizes the right moment, and neighbor filtering supplies the surrounding context relational queries need.
(a) The event key precisely matches the query, enabling retrieval of the exact target segment.
(b) Both settings retrieve the correct segment, but neighbor filtering supplies the ±Δ context needed to resolve relational queries that a single clip cannot answer.
BibTeX
Please cite MERIT if you find this work useful.
@inproceedings{choi2026merit,
title = {Keep It Simple: Multi-Key Episodic Memory Retrieval
for Ultra-Long Video Understanding},
author = {Choi, Yeeun and Yoo, Youngbeom and Lee, Joon-Young
and Kang, Hyolim and Kim, Seon Joo},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026},
eprint = {2608.07663},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}
