PoVi

PoVi Agentic Poetry-to-Video Generation with Knowledge Grounding and Figurative Understanding

View the stories Framework Citation

诗中有画
画中有诗苏轼

Visual stories

Between the lines, a world comes into view.

Three classical works, each told first through its lived context and then line by line. Frames are taken from PoVi’s generated teaching videos.

05 · Mooring at night 05 / 09

Tang poetry · Introductory

Mooring by Maple Bridge

Mooring by Maple Bridge · Full film Coming soon

The challenge

A poem is not a visual specification.

Classical poetry compresses people, events, cultural references and emotions into a few lines. Rendering the visible objects alone can produce beautiful imagery while losing what the poem means.

PoVi reconstructs the implicit context and source–target relations first. That understanding becomes an evidence-bounded plan for narration, visual direction and synchronized audiovisual generation.

  1. The input

    A few lines. A whole world.

    A poem leaves its people, histories and figurative relations partly unstated.

  2. The missing step

    Make the implicit explicit.

    Recover context, connect metaphor to meaning, and preserve interpretive uncertainty.

  3. The output

    A story to hear. A world to see.

    Direct coherent scenes and synchronize their timing with spoken narration.

Framework

Ground the meaning. Then direct the scene.

01 · Understand

Evidence-bounded planning

A semantic director graph connects context, causal relations, metaphor and visual constraints. Claims cannot exceed the authority of their supporting evidence.

02 · Direct

Role-separated realization

Five specialized stages create the background story, narration, close reading, recurring characters and shot descriptions through validated artifacts.

03 · Synchronize

Audio-clocked assembly

Measured speech duration sets the final clock. Shot allocations, clips, narration and transitions are aligned to the synthesized audio.

The semantic director graph

Each stage receives a restricted graph view and communicates through validated artifacts.

  • Historical context
  • Causal relations
  • Metaphor mappings
  • Visual constraints
01

Background story

Causal context and evidence become a continuous story.

02

Narration

The background story is segmented into narration and visual events.

03

Close reading

The story and metaphor nodes inform a line-by-line explanation.

04

Character assets

The merged narration identifies recurring people and their appearance.

05

Visual direction

Narration, metaphor nodes and character assets guide shots; events remain fixed.

Speech synthesisGenerated shotsA narration-aligned film

Stages 02 and 03 both use the background story. A program merges their outputs before Stage 04; Stage 05 receives the merged narration and character assets. Each stage runs in a fixed sequence.

Evidence before certainty

A recalled claim is not a documented fact.

Schema, reference and grounding checks validate the graph. Model-recalled knowledge remains plausible or contested; consensus requires high-authority evidence. Failed checks trigger directed repair.

  1. LowModel recollectionPlausible / contested
  2. MediumPoem & metadataNeeded for “documented”
  3. HighScholarly evidenceNeeded for “consensus”

Authority sets an admissibility threshold, not proof of truth or consensus. The evidence inventory is fixed before planning; validation checks citations, structure and line coverage.

Measured speech is the clock

The voice sets the pace of the film.

Provisional shot durations are rescaled after speech synthesis. The final visual sequence follows the measured audio track.

Synthesized narration 12.0 s
Rescaled visual sequence
Shot 013.6 s Shot 022.4 s Shot 036.0 s

Example allocation: 3 : 2 : 5 → 3.6 : 2.4 : 6.0 seconds

PoVi-Bench

Measure the meaning, not just the picture.

Works
120

60 Tang poems and 60 Song ci

Metrics
17

Across three evaluation dimensions

Annotation groups
5

Frozen gold annotations, reserved for evaluation

Required information, metaphor maps, evidence claims, ordered background events and visual requirements separate what a system understands from what it actually realizes. Gold annotations are reserved for evaluation.

Both literary forms are balanced across three levels of interpretive difficulty.

Composition of PoVi-Bench by literary form and difficulty.
Literary form Easy Medium Challenging Total
Tang poetry 20 20 20 60
Song ci 20 20 20 60
Total 40 40 40 120
01 · Semantic grounding

Is the poem
understood?

S1
Information coverage
S2–S4
Metaphor source, target & bridge
S5
Contextual support
S6
Interpretive calibration
02 · Discourse coherence

Does meaning
become explanation?

D1
Narrative structure
D2
Prelude–explication linkage
D3
Modality complementarity
D4
Audiovisual pacing
03 · Audiovisual realization

Does the film
carry the meaning?

R1–R2
Visual coverage & correctness
R3–R4
Quality & character continuity
R5
Visual metaphor
R6–R7
Correspondence & synchrony
Explore all 17 metric definitions ＋

Seventeen atomic metrics distinguish understanding, explanation and audiovisual realization.

The 17 primary metrics of the benchmark, grouped by the three challenges of the task.
ID Metric Scale What it asks Gold field
S1 Information Coverage [0,1] Is the information a faithful explanation must convey actually conveyed? required_info
S2 Metaphor Source [0,1] Is the vehicle imagery explicitly identified? metaphor_maps
S3 Metaphor Target [0,1] Is the abstract referent stated without over-generalization? metaphor_maps
S4 Mapping Bridge [0,1] Is it explained why the source licenses the target? metaphor_maps
S5 Metaphor Contextual Support [0,1] Is the figurative reading supported by text, context or tradition rather than free association? metaphor_maps
S6 Interpretive Calibration [0,1] Does the epistemic stance match the annotated status of the reading? status
D1 Narrative Structure 1–5 Are event order, state change, causality and scene transitions correct? key_events
D2 Prelude–Explication Linkage 1–5 Is the background dramatized concretely and does it lead into line-by-line explication? key_events
D3 Modality Complementarity 1–5 Does the visual channel add information rather than duplicate the narration? —
D4 Audiovisual Pacing 1–5 Are shot duration, transition frequency and information density coherent across phases? —
R1 Visual Requirement Coverage [0,1] Do the required visual elements appear at all? visual_req
R2 Visual Semantic Correctness [0,1] Is what is shown correct in identity, period, space, action and outcome? visual_req
R3 Static Visual Quality 1–5 Sharpness, exposure, colour, texture, subject legibility. —
R4 Character Identity Continuity [0,1] Is the same narrative character still the same person across shots? —
R5 Visual Metaphor Realization 1–5 Are source, bridge and target realized visually? metaphor_maps
R6 Cross-modal Correspondence [0,1] Do concurrent narration and image refer to the same content? —
R7 Narration–Event Synchrony [0,1] Does each key event appear when it is narrated? key_events

Results · human ratings

The biggest gains lie in what the poem leaves unsaid.

All 120 works are evaluated with three planning backbones and no external reference material (B = empty); gold annotations are used only for evaluation. The single-call comparisons hold the backbone, video generator and speech synthesizer fixed. Human ratings are the primary evidence.

Single call vs. PoVi

Single call PoVi

  • S1 · Information coverageGemini-3.5-Flash

    0.611 → 0.814

  • S1 · Information coverageSeed-2.0-Lite

    0.625 → 0.859

  • S3 · Metaphor targetDeepSeek-V4-Flash

    0.512 → 0.775

  • S5 · Metaphor contextual supportDeepSeek-V4-Flash

    0.688 → 0.954

  • D2 · Prelude–explication linkageGemini-3.5-Flash

    0.633 → 0.800

  • R1 · Visual coverageWan2.2-VACE

    0.671 → 0.832

  • D2 · Prelude–explication linkageDeepSeek-V4-Flash · a decline

    0.825 → 0.569

Selected metrics from the human-rated tables below, including the one notable decline. Narration metrics compare planning backbones; R1 uses the Gemini planner. The axis starts at 0.4.

The largest narration gains concern information and figurative meaning that the poem leaves implicit.

Narration metrics, human ratings. D1 and D2 are 1--5 scores divided by 5.
Backbone System S1 S2 S3 S4 S5 S6 D1 D2
Gemini-3.5-Flash single call 0.611 0.931 0.851 0.617 0.889 0.943 0.815 0.633
Gemini-3.5-Flash ours 0.814 0.951 0.915 0.789 0.943 0.965 0.952 0.800
DeepSeek-V4-Flash single call 0.533 0.756 0.512 0.334 0.688 0.955 0.725 0.825
DeepSeek-V4-Flash ours 0.706 0.894 0.775 0.521 0.954 0.957 0.897 0.569
Seed-2.0-Lite single call 0.625 0.478 0.400 0.365 0.665 0.944 0.906 0.334
Seed-2.0-Lite ours 0.859 0.692 0.701 0.428 0.907 0.961 0.952 0.688

Explicit planning improves or preserves every reported full-pipeline visual score across four generators.

Video metrics, human ratings, Gemini planner. R3 and R5 are 1--5 scores divided by 5.
System R1 R2 R3 R4 R5
MiniMax-H3
single call 0.870 0.910 0.866 0.950 0.882
ours 0.930 0.930 0.918 0.950 0.912
w/o ref. img. 0.812 0.796 0.840 0.000 0.870
Wan2.2-VACE
single call 0.671 0.788 0.698 0.900 0.734
ours 0.832 0.876 0.898 0.900 0.824
w/o ref. img. 0.652 0.825 0.810 0.000 0.810
Veo-3.1
single call 0.699 0.980 0.897 0.950 0.811
ours 0.754 0.998 0.944 0.984 0.844
w/o ref. img. 0.534 0.982 0.880 0.000 0.830
Seedance-2.0
single call 0.770 0.958 0.939 0.880 0.896
ours 0.862 0.978 0.984 0.978 0.903
w/o ref. img. 0.765 0.974 0.810 0.000 0.880

R1: visual coverage · R2: semantic correctness · R3: static quality · R4: character continuity · R5: visual metaphor. “w/o ref. img.” is the reference-image ablation.

Cross-modal timing and agentic-pipeline comparison ＋

Matching the narrated content is easier than making each event appear at exactly the right time.

Cross-modal metrics under our pipeline, human ratings. D3 and D4 are 1--5 scores divided by 5.
Generator R6 R7 D3 D4
Wan2.2-VACE 0.881 0.812 0.785 0.878
MiniMax-H3 0.904 0.872 0.919 0.947

In this separate agentic-pipeline comparison, PoVi improves background coverage and the transition into close reading.

Comparison with agentic video pipelines on DeepSeek-V4-Flash, human ratings. D1, D2 and R3 are 1--5 scores divided by 5.
System S1 S2 S3 S4 S5 S6 D1 D2 R1 R2 R3 R4
ArcReel 0.780 1.000 0.980 0.900 0.800 0.840 1.000 0.640 0.800 0.880 0.920 0.900
Lumenx 0.700 0.940 0.780 0.680 0.960 0.900 0.980 0.680 0.780 0.880 0.880 1.000
ours 0.880 0.940 0.920 0.930 0.970 0.960 1.000 0.880 0.880 0.900 0.960 1.000

The manuscript reports different PoVi scores in the narration and agentic-pipeline comparisons, but does not explain the evaluation conditions behind that difference.

Automatic evaluation and human agreement ＋

Automatic evaluation complements the human-rated experiments; it is reported separately.

Narration metrics, Gemini-3.5-Flash judge; human ratings appear in the main results above.
Backbone System S1 S2 S3 S4 S5 S6 D1 D2
Gemini-3.5-Flash single call 0.729 0.976 0.929 0.670 0.941 0.943 0.978 0.823
Gemini-3.5-Flash ours 0.914 0.983 0.981 0.953 0.980 0.979 0.981 0.954
DeepSeek-V4-Flash single call 0.634 0.960 0.863 0.816 0.898 0.966 0.933 0.806
DeepSeek-V4-Flash ours 0.906 0.976 0.954 0.934 0.952 0.914 0.983 0.921
Seed-2.0-Lite single call 0.464 0.523 0.500 0.202 0.694 0.823 0.912 0.370
Seed-2.0-Lite ours 0.821 0.758 0.724 0.524 0.922 0.965 0.969 0.651

Automatic evaluation complements the human-rated experiments; it is reported separately.

Video metrics, Gemini-3.5-Flash judge. R3 and R5 are 1--5 scores divided by 5; human ratings appear in the main results above.
System R1 R2 R3 R4 R5
MiniMax-H3
single 0.811 1.000 0.985 1.000 0.821
ours 0.831 1.000 0.988 1.000 0.874
Wan2.2-VACE
single 0.682 0.880 0.938 1.000 0.800
ours 0.790 0.881 0.935 1.000 0.848
w/o ref. img. 0.689 0.813 0.908 0.000 0.807
Veo-3.1
single 0.731 0.998 0.973 1.000 0.830
ours 0.779 1.000 0.999 1.000 0.863
Seedance-2.0
single 0.820 1.000 0.995 1.000 0.888
ours 0.824 1.000 1.000 1.000 0.886

Automatic evaluation complements the human-rated experiments; it is reported separately.

Cross-modal metrics, Gemini-3.5-Flash judge. D3 and D4 are 1--5 scores divided by 5; human ratings appear in the main results above.
Generator R6 R7 D3 D4
Wan2.2-VACE 0.890 0.817 0.937 0.902
MiniMax-H3 0.940 0.925 0.968 0.975
Seedance-2.0 0.933 0.897 0.992 0.986

Automatic evaluation complements the human-rated experiments; it is reported separately.

Comparison with agentic video pipelines, Gemini-3.5-Flash judge; human ratings appear in the main results above.
System S1 S2 S3 S4 S5 S6 D1 D2 R1 R2 R3 R4
ArcReel 0.830 0.980 0.934 0.972 0.944 0.800 1.000 0.560 0.767 1.000 0.988 0.900
Lumenx 0.818 0.934 0.894 0.760 0.934 0.860 1.000 0.440 0.702 1.000 1.000 1.000
Ours 0.906 0.976 0.954 0.934 0.952 0.914 0.983 0.921 0.839 1.000 1.000 1.000

Moderate agreement supports retaining human ratings as the primary evidence.

Agreement between Gemini-3.5-Flash and human ratings on the eight narration metrics over all 120 works in PoVi-Bench.
Evaluator SRCC ↑ KRCC ↑ PLCC ↑ nRMSE ↓
Gemini-3.5-Flash 0.505 0.429 0.503 0.162

Scope & limitations

A richer process. A measurable cost.

PoVi uses eight initial model calls per poem, compared with one for the baseline. The 120-work benchmark cannot span the full historical and formal diversity of classical Chinese poetry.

Gains are not uniform: DeepSeek’s prelude–explication linkage (D2) falls from 0.825 to 0.569. The automatic evaluator shows moderate human agreement and saturation on some visual metrics. The supplied draft does not specify human-rater count, background or inter-rater agreement.

Eight initial model calls establish the cost of structured planning before any repairs.

Model calls per poem. A run with no repairs uses eight calls; the worked example used twelve.
Stage First call Max repairs
Knowledge inventory 1 0
Graph generation 1 3
Context story 1 3
Story narration 1 3
Close-reading planner 1 2
Close-reading realizer 1 2
Character assets 1 2
Visual direction 1 2
Total 8 17

Citation

Paper title

PoVi: Agentic Poetry-to-Video Generation with Knowledge Grounding and Figurative Understanding

Author names, venue and publication details are not confirmed in the supplied manuscript.

Enlarged paper figure