Gemma 4 video understanding

Two videos were analysed by Gemma 4 12B running locally: a screencast and a meeting clip. Only the written results are shown here, not the videos or stills from them.

Names, contact details and the meeting's title have been removed from the meeting clip's results and are marked in square brackets.

Screencast: NotebookLM importing ten sources

Model
Gemma 4 12B (GGUF Q4_K_M, LM Studio, fully local)
Run
Local: LM Studio on an Apple M4 Max with 128 GB
Length
25.8 s
Method
26 frames @ 1 FPS, 1024px wide; two passes (description + scene breakdown); scene boundaries verified frame-by-frame by Claude Opus 4.8

Scenes

  1. 0 s to 1 s
    Source discovery

    The notebook opens on NotebookLM's source-discovery results for the query “What are AI use policies are published…”. Ten sources on university AI-use policies are pre-selected, with the Import button ready. The Chat pane still shows the “Let's start your notebook…” welcome.

    On screen: 10 sources selected · Import · Let's start your notebook…

  2. 1 s to 7 s
    Sources imported, notebook created

    Import lands: a “10 sources added” toast confirms, the notebook home shows the title “Russel Group AI Use · 10 sources · 3 Jun 2026”, and a tooltip invites customising the notebook with a cover image. Source entries finish resolving in the left panel.

    On screen: 10 sources added · Add a cover image and custom note to personalise your notebook · 10 sources · 3 Jun 2026

  3. 7 s to 10 s
    Overview summary starts generating

    The Customise tooltip clears and the Chat pane begins composing the notebook overview — skeleton placeholder bars appear where the summary text will land.

  4. 10 s to 14 s
    Studio tour: data table tooltip

    Attention moves to the Studio panel. Hovering the Data tool raises the tooltip “Generate a data table from your sources”; the tile highlights as it is selected.

    On screen: Generate a data table from your sources

  5. 14 s to 24 s
    Data table generating

    The Data tool is activated: Studio shows “Generating data table… based on 10 sources”. The overview summary in Chat is still composing — both generations run in parallel.

    On screen: Generating data table… · based on 10 sources

  6. 24 s to 25.8 s
    AI overview appears

    The Chat pane renders the finished overview: academic integrity policies and misconduct procedures for Imperial College London, LSE, and the University of Manchester — academic malpractice, generative AI, contract cheating, plagiarism, and rules for referencing AI usage. The data table is still generating as the recording ends.

    On screen: academic integrity policies · misconduct procedures · referencing AI usage

Raw model output

Pass 1 · Open description (8,200 total / 944 completion)
A screen recording shows a user interacting with NotebookLM, an AI-powered research and note-taking tool.

The notebook is titled “Russel Group AI Use” and focuses on academic integrity policies related to generative AI across several UK universities.

1. Initial Setup: The user starts a new notebook. A “Chat” window displays a welcoming message, inviting the user to start by asking questions or creating something based on their sources. On the left, “Source discovery” shows search results related to AI use policies at universities.
2. Adding Sources: The user imports several specific documents and web pages. These include articles regarding academic integrity, generative AI guidelines, and official university policy documents. A tooltip appears over a customization button, encouraging the user to personalize their notebook with a cover image.
3. Exploring Tools: As sources are confirmed as added (showing “10 sources”), the interface highlights various features in the “Studio” panel on the right. Interactive tooltips appear for different content formats: “Generate an AI slide deck…”, “Infographic - Generate an AI infographic based on your sources,” and “Generate a data table from your sources.”
4. Summary Generation: The “Chat” window updates with a summary generated by the AI. It synthesizes information about academic integrity policies and misconduct procedures for institutions like Imperial College London, the London School of Economics (LSE), and the University of Manchester. The text highlights rules regarding plagiarism, contract cheating, and the requirement for students to produce original work while properly citing AI usage.
Pass 2 · Scene breakdown (timings later corrected by frame review) (9,867 total / 2,603 completion)
Scene 1 (0–2s) — A search results page on NotebookLM titled “Russel Group AI Use,” showing several source links related to university AI policies. The user is browsing through the discovered sources in the initial setup phase of a notebook.

Scene 2 (2–4s) — A transition to the main “Chat” interface where a folder icon and the title “Russel Group AI Use” appear, with a notification saying “10 sources added.” The user clicks the “Import” button, and the system processes the selected source materials into the active workspace.

Scene 3 (4–6s) — The “Studio” panel on the right side of the screen with various tool icons like Video, Report, Quiz, and Data. The user explores the interactive features by hovering over different tools, seeing informative tooltips for generating video slides and data tables.

Scene 4 (6–10s) — A text block in the center “Chat” area providing a summary of academic integrity policies from Imperial College London, LSE, and the University of Manchester. The AI processes the selected source material to generate a cohesive summary for the user.

Notes on the method and the result

  • 26 frames @ 1 FPS, 1024px wide, via `lmstudio_video.py` → LM Studio `/v1/chat/completions`.
  • Pass 1 prompt: "This is a screen recording. Describe what application and content is shown, what the user does, and how the screen changes over time. Read out any prominent on-screen text, titles, or headings you can see." (8,200 tokens total)
  • Pass 2 prompt: "This is a screen recording of a web app. Break it into scenes. For each scene give the approximate start time in seconds (frames are 1 per second), one line on what is visible, and one line on what changes or what the user does." (9,867 tokens total)
  • Scene boundaries in the deliverable were verified frame-by-frame by Claude Opus 4.8 (the model's own Pass-2 timings ran compressed — content accurate, timing unreliable).
  • Strong: on-screen text reading (notebook title, tooltips, toasts, source counts, university names), workflow comprehension, temporal ordering.
  • Weak: absolute timestamps (compressed ~26s into ~10s); needs external frame review or labelled-frame prompting for timing-critical work.

A meeting clip

Model
Gemma 4 12B (visuals) + Parakeet TDT 0.6B (speech), both fully local
Run
Local: LM Studio on an Apple M4 Max with 128 GB
Length
30 s
Method
First 30s of a 2h Teams recording. Visuals: 30 frames @ 1 FPS, 1024px, Gemma 4 12B via LM Studio. Speech: Parakeet TDT 0.6B v3 (MLX) timestamped transcript. Scene boundaries verified frame-by-frame by Claude Opus 4.8.

Scenes

  1. 0 s to 2 s
    Teams recording title card

    The recording opens on the dark Microsoft Teams title card for the session: “[session title removed]”, dated [date removed], recorded by [name removed], organised by the [organiser removed].

    On screen: [session title removed] · [date removed]

  2. 2 s to 16 s
    Title slide, welcome

    The shared screen switches to the opening slide: a robot figure over [place removed] rooftops with “[session title removed]” on a dark blue panel, [organisation removed] and [organisation removed] logos top-left, contact email below. The Teams participant gallery lines the right edge. The presenter welcomes everybody to the session.

    On screen: [session title removed] · [email removed]

  3. 16 s to 17 s
    Section slide: getting to know each other

    A brief interstitial slide — “Getting to know each other” on a plain light background — lands exactly as the presenter says he wants to start by getting to know each other.

    On screen: Getting to know each other

  4. 17 s to 30 s
    Presenter introduction slide

    The presenter slide appears: [name removed], [job title removed] at the [organisation removed], with email and [website removed], alongside the [lab name removed] graphic. The presenter introduces his role and starts on his background as the clip ends.

    On screen: [name removed] · [job title removed] · [lab name removed]

Speech transcript

  • 1.9–10.6 s Welcome everybody to this session on [session title removed].
  • 10.6–16.6 s And I want to start by getting to know each other.
  • 16.6–26.6 s And so a little bit above me, uh, I have uh I'm an [job title removed] at the [organisation removed].
  • 26.6–29.7 s But before this, I ran a uh

Raw model output

Gemma 4 12B · visual description pass (8,698 total / 954 completion)
The video starts with an opening title screen for a session titled “[session title removed]”, scheduled for [date removed], recorded by [name removed] and organized by the [organiser removed].

The scene then transitions to a presentation slide featuring a robot illustration and large text on a dark blue background that reads: “[session title removed].” The [organisation removed] and [organisation removed] logos are visible in the top-left corner, and an email address for [name removed] is displayed at the bottom. On the right side of the screen, a Microsoft Teams gallery shows several other participants' video thumbnails and names, including [name removed] and [name removed].

A short transition occurs where the slide text changes to “Getting to know each other.”

The final part of the sequence shows a stable slide introducing the presenter: “[name removed],” with his email ([email removed]), title as “[job title removed]” at the “[organisation removed],” and his website, [website removed], at the bottom. To the right of this text is a detailed graphic for the “[lab name removed],” which outlines their mission to provide tools and strategies for academic productivity through various resources like AI tools and writing tips. The side panel with other participants remains visible throughout these frames.
Parakeet TDT 0.6B · speech transcription (raw SRT segments) (4 segments, 0–29.7s)
[00:01.9 → 00:10.6] Welcome everybody to this session on [session title removed].

[00:10.6 → 00:16.6] And I want to start by getting to know each other.

[00:16.6 → 00:26.6] And so a little bit above me, uh, I have uh I'm an [job title removed] at the [organisation removed].

[00:26.6 → 00:29.7] But before this, I ran a uh

Notes on the method and the result

  • Clip: `ffmpeg -t 30` re-encode to 1280px / CRF 26 / AAC 64k mono (478 KB).
  • Visuals: 30 frames @ 1 FPS, 1024px → `lmstudio_video.py` → LM Studio. Prompt: "This is the opening of a recorded online meeting or webinar. Describe what is shown on screen: the application, layout, any slides or shared content, visible people, and how the screen changes over time. Read out any prominent on-screen text…" (8,698 tokens)
  • Speech: `speech-to-text` skill, `parakeet_to_srt.py` → 4 SRT segments.
  • Scene boundaries verified frame-by-frame by Claude Opus 4.8.
  • Gemma read the Teams title card metadata, slide titles, presenter details, and even participant names from the gallery thumbnails ([name removed], [name removed]).
  • Parakeet's timestamps and Gemma's visual scenes were produced independently and align naturally (slide change at ~16s lands on "getting to know each other").
  • Speech lane is verbatim incl. disfluencies ("a little bit above me, uh") — fine for sync, run cleanup prompts from the speech-to-text skill for publishable text.