Multimodal AI is becoming the default interface
Speaking, showing, uploading and pointing are converging into one assistant experience. The interface feels simpler because the engineering underneath has become more complicated.
What is happening?
Multimodal AI lets a person type a question, speak, show the camera, upload a document or combine several of those inputs in one conversation. That is useful because people do not experience the world as text alone. It is also easy to overestimate. The model may receive a resized image, a sparse set of video frames, downsampled audio or a rendered PDF rather than the full original evidence. A dependable product records those transformations, checks conflicts between modalities, protects recordings and provides equivalent text, captions or descriptions for people who cannot use the preferred input or output mode.
Why this trend is moving
- 01End-to-end and connected multimodal architectures can carry information across text, vision and audio without forcing every interaction through a transcript.
- 02Low-latency streaming APIs make interruption, live translation, camera input and spoken tool use practical in ordinary applications.
- 03Document, image, audio and video understanding are being consolidated into general model APIs instead of separate specialist products.
- 04Open-weight omni models make multimodal experimentation possible outside a small group of hosted providers.
- 05New benchmarks and security research are exposing cross-modal failures, temporal weaknesses and prompt-injection paths that text-only evaluation misses.
What this means in practice
- The product should document which media is processed directly and which is converted through OCR, transcription, frame sampling or other specialist services.
- Resolution, duration, sampling and conversation history should be treated as explicit quality, latency and cost settings.
- Claims should remain linked to pages, image regions, speakers and timestamps so a reviewer can inspect the evidence.
- Conflicting audio, visual and textual evidence should trigger a visible warning, clarification or abstention rather than a silent compromise.
- Accessibility requires equivalent information across text, speech, captions, transcripts, descriptions and non-voice controls.
- Every parsed modality can carry sensitive data or adversarial instructions, so authorization and consequential actions must remain outside the model.
What the headline leaves out
This is the practical technical view: how the system is put together, where it can fail, and what a real deployment asks from the team running it.
How it is built
A production multimodal system starts by defining the task, capture boundary and consent state. It normalizes files and streams, runs modality-specific perception, converts selected detail into tokens or embeddings, aligns media by time and meaning, reasons over the combined evidence, and renders an accessible response. Verification and telemetry preserve preprocessing settings, evidence references, versions, latency and retention state. Some products use one end-to-end omni model; others orchestrate OCR, speech, vision and language components.
How inference behaves
Images may be resized or divided into tiles, audio may be resampled and tokenized by duration, documents may be rendered page by page, and video may be represented by sampled frames plus audio. Encoders project those signals into a shared space or token sequence where a transformer can connect them. Live systems add stateful streaming, voice activity detection, interruption, clock synchronization and output synthesis. The model can only reason over the detail that survives this pipeline.
What the tests can miss
A credible test suite measures OCR, speech recognition, diarization, spatial reasoning, document structure, temporal understanding and non-speech audio separately, then adds tasks that genuinely require evidence from more than one modality. It should vary resolution, frame rate, noise, language and missing inputs; test contradiction handling and calibration; measure live turn behavior and accessibility equivalence; and attack images, documents, audio and physical scenes with injected instructions.
What deployment involves
Choose the architecture from the evidence requirement. General assistants can use a unified hosted model. Documents and regulated workflows often benefit from inspectable specialist stages. Mobile and wearable products can redact or compress on device before cloud reasoning. High-consequence uses should preserve original media, derived artifacts, versions and human approval. Progressive analysis—a coarse pass followed by targeted high-resolution work—usually controls cost better than processing everything at maximum detail.
Where the risks sit
Multimodal systems collect faces, voices, locations, background conversations, screens and documents, often from people who did not initiate the session. Hidden or visible instructions can also arrive through screenshots, PDFs, signs, subtitles, audio and video. Clear capture controls, minimization, redaction, retention limits, untrusted-content isolation, constrained tools and external authorization are essential. A face or voice resemblance must not become authentication simply because the model can describe it.
What it really costs
Media creates cost through upload, storage, decoding, preprocessing and modality-specific tokenization before answer generation begins. Long audio, high-resolution images, dense video sampling and accumulated live-session history can dominate the bill and increase tail latency. Task-aware cropping, frame selection, transcript reuse, progressive analysis, context compression and modality routing reduce expense when quality is measured after each change.
What the evidence supports
The product direction is clear. OpenAI, Google and Qwen expose systems that combine speech, vision, text and tool use, and official documentation describes live streaming and media-processing controls. Research on shared embeddings and omni architectures shows several viable designs rather than one settled approach. At the same time, MMMU-Pro, MMAU and Video-MME-v2 report difficult visual, audio and temporal reasoning gaps, while OWASP and recent research demonstrate multimodal prompt-injection risks. The evidence supports multimodal interfaces as a durable product direction, not claims of complete human-like perception.
How it works in practice
A single assistant can accept text, images, audio, video and documents, but those inputs do not arrive as equal evidence. Each modality is sampled, compressed, tokenized and aligned differently. Reliable products preserve provenance, make information loss visible, test cross-modal consistency and keep consent, accessibility and security outside the model.
How the parts work together
The headline technology is only one part of the product. Reliability, security and cost are usually decided by the handoffs around it.
- 01
Task, consent and capture boundary
The application defines what the user is asking, which sensors or files are allowed, whether recording is active, and which data may be retained. A microphone or camera should not become an open-ended background input.
- 02
Acquisition and normalization
Images are rotated and resized, documents are rendered, audio is resampled, video is decoded and timestamps are preserved. The system records the original file and the derived representation used for inference.
- 03
Modality-specific perception
Vision encoders, OCR, speech recognition, diarization, audio encoders and frame samplers extract usable signals. Some products expose these stages; others perform them inside one hosted model.
- 04
Tokenization and compression
Media becomes tokens or embeddings under a finite context and cost budget. Resolution, frame rate, crop strategy and audio duration determine which details survive.
- 05
Temporal and semantic alignment
The system connects words, speakers, frames, objects and timestamps. Alignment errors can make a correct transcription refer to the wrong person or a visual event appear to happen at the wrong time.
- 06
Reasoning, retrieval and tools
The model combines the selected evidence, may call tools, and produces a structured answer or action proposal. Conflicting modalities should trigger clarification or abstention rather than silent averaging.
- 07
Accessible output rendering
The response is delivered as text, speech, captions, highlighted regions, timestamps or structured data. Users need equivalent information when they cannot see, hear or interact with the preferred output mode.
- 08
Verification, telemetry and retention
The application records model and preprocessing versions, modality usage, latency, failures, consent state and evidence references. Raw media, transcripts and derived embeddings follow explicit access and deletion rules.
Estimate the limits before the demo
These equations are planning tools rather than substitutes for testing. They help expose a design that is unlikely to fit its hardware, budget, reliability or risk limits.
Media cost begins before reasoning
input budget ≈ text tokens + image-tile tokens + audio seconds × audio-token rate + sampled frames × frame-token rate Exact accounting is provider- and model-specific. The formula is useful because it exposes the levers: image resolution, video sampling, audio duration and conversation history can dominate the request before the model writes one word.
- A high-resolution receipt may require several visual tiles while a thumbnail uses one.
- A one-hour recording can consume far more context than its final summary.
- Lowering video frame rate saves cost but can remove a brief event that matters.
Live interaction has a complete latency path
turn latency ≈ capture buffer + network + preprocessing + model time + output synthesis and playback Optimizing only model generation does not guarantee a natural conversation. Voice activity detection, upload jitter, tool calls and audio playback all influence whether the assistant interrupts, hesitates or talks over the user.
- A shorter audio buffer responds faster but may cut off the end of a phrase.
- A vision lookup during a live call can add a noticeable pause even when speech generation is fast.
- Streaming partial output reduces perceived delay but makes correction harder after the model commits.
Cross-modal reliability is a chain
usable result depends on perception × alignment × reasoning × output fidelity This is a planning model, not an independence claim. Errors are often correlated. A correct answer still fails the user when OCR missed a decimal, the voice was assigned to the wrong speaker, or the spoken response omitted a qualification present in the text.
- Correct speech recognition attached to the wrong participant produces a false meeting record.
- Accurate frame descriptions cannot answer an event-order question when timestamps were lost.
- A correct visual answer is inaccessible when no textual alternative is provided.
One conversation can hide several very different systems
From the user side, multimodal AI looks simple: speak, point the camera, upload a document and ask a question. Underneath, the product may be a chain of specialist services. Speech becomes text, selected video frames become images, OCR extracts document text, and a language model receives the combined result. Another product may use an end-to-end omni model that consumes audio and vision directly.
Neither design is automatically superior. Cascaded systems make intermediate results visible and allow teams to replace one component. They also accumulate handoff errors and latency. Unified models can preserve tone, timing and cross-modal context that a transcript loses, but their internal mistakes are harder to isolate and the whole stack may change with one model update.
The engineering question is therefore not whether a model is multimodal. It is where perception happens, what representation crosses each boundary and which evidence can be inspected when the answer is wrong.
- Document the real pipeline rather than the marketing label.
- Retain timestamps, page numbers, regions and speaker identifiers through every transformation.
- Expose intermediate evidence for workflows that require review.
- Use specialist tools when their accuracy or control is measurably better.
Shared space, separate encoders and end-to-end models solve different problems
A common architecture gives each modality a specialist encoder, then projects its output into a language model. ImageBind demonstrated a shared embedding space across image, text, audio, depth, thermal and motion data. Other systems place visual and audio tokens directly beside text tokens and train a transformer to reason across the sequence.
Omni models add generation. GPT-4o was described as one network trained across text, vision and audio. Qwen2.5-Omni uses audio and visual encoders, time-aligned position information and a Thinker-Talker design that produces text and streaming speech. These examples show convergence at the interface while retaining architectural differences inside.
A shared representation improves cross-modal retrieval and conversation, but it can also blur boundaries. A voice characteristic may influence a judgment that should depend only on words. A diagram may override correct text. Production systems need explicit rules about which modality is authoritative for each field.
- Use shared embeddings for retrieval and similarity tasks.
- Use token-level fusion when relationships between details and timing matter.
- Keep deterministic fields such as totals, dates and identifiers independently validated.
- Test whether irrelevant visual or acoustic features change decisions.
Sampling and compression decide the evidence before reasoning begins
A model cannot recover a detail that the input pipeline discarded. Images may be resized or split into tiles. PDF pages may be rendered at a fixed resolution. Audio may be downsampled or mixed to one channel. Video services often sample frames rather than process every frame at full resolution.
These choices are reasonable engineering compromises. They become dangerous when the product hides them. A one-frame-per-second video path can miss a brief hand movement, status light or on-screen code. A low-resolution image can turn 8 into 3. Mono audio can remove channel information that identified where a sound came from.
The application should match preprocessing to the task. Reading tiny labels needs higher visual resolution than describing the scene. Detecting a rapid event needs denser temporal sampling than summarizing a lecture. Long files may need a coarse pass followed by targeted reprocessing of relevant pages or time ranges.
- Store preprocessing settings with the result.
- Use task-aware crops, pages and time windows.
- Escalate uncertain details to higher resolution or a specialist model.
- Never describe a sampled summary as a complete inspection of the original media.
A voice assistant must manage timing, interruption and state
Live multimodal systems are stateful. Audio arrives continuously, video may arrive as periodic frames, and the model may begin responding before the user finishes. The service has to infer turn boundaries, support interruption and decide how much recent media remains in context.
WebSockets or WebRTC reduce transport delay, but they do not remove the product choices. Voice activity detection may stop too early for a hesitant speaker. A long silence can mean the user is thinking, not finished. An interrupted response may have already called a tool. Audio and video clocks can drift.
A safe implementation separates conversational state from action state. Interrupting speech should stop playback immediately, while a consequential tool action needs its own cancellation and confirmation rules. Session compression should preserve decisions and user preferences without retaining unlimited raw recordings.
- Measure end-of-turn errors as well as average latency.
- Preserve tool and approval state outside the audio conversation.
- Synchronize media with monotonic timestamps.
- Give users visible microphone and camera controls plus an immediate stop action.
The system needs a policy for conflicting evidence
A slide may say one amount while the speaker says another. A chart can disagree with its surrounding paragraph. Background audio may contradict a caption. Multimodal systems are valuable because they can notice these conflicts, yet they can also resolve them silently and choose the wrong source.
The product should define authority by task. For a signed form, the visible field and document version may be primary. For a meeting, the audio and approved transcript may be primary while the model-generated summary remains derived. For a safety inspection, a sensor reading may outrank a visual estimate.
Answers should preserve uncertainty and provenance. Instead of reporting one number, the system can say that the spoken value differs from the slide, provide timestamps and ask which version governs. This is more useful than a fluent compromise that no source actually contained.
- Tag every extracted claim with modality, location and timestamp.
- Define authoritative sources for each business field.
- Surface contradictions instead of averaging them away.
- Ask for clarification when the conflict changes the decision.
Each sense can pass alone while the combined task still fails
Image, audio and video benchmarks measure different capabilities. MMMU-Pro removes many questions that can be answered without looking at the image. MMAU tests speech, environmental sound and music reasoning. Video-MME-v2 emphasizes visual aggregation, temporal dynamics and consistency across related questions. These tests expose gaps that a general text score cannot.
Production evaluation should begin with the actual media distribution: camera angles, document templates, accents, languages, noise, screen recordings and device quality. It should then add cross-modal cases where the answer requires two inputs at once, as well as conflict cases where one modality is misleading or missing.
Human review also needs a clear reference. A vague question such as “understand this meeting” cannot produce a stable benchmark. Define whether success means accurate transcription, speaker attribution, decision extraction, accessibility output or a correct downstream action.
- Report performance separately by modality and media quality.
- Test cross-modal dependency rather than allowing a text-only shortcut.
- Measure timestamps, regions and speaker attribution, not only final-answer accuracy.
- Include abstention, disagreement and missing-modality cases.
A multimodal interface can remove barriers or create new ones
Voice and vision can make software easier to use for people who cannot type, read small text or navigate a complex screen. The same interface can exclude users when essential information exists only in speech, only in an image, or only through a gesture.
W3C guidance treats captions, transcripts and descriptions of visual information as distinct accessibility needs. Automatic generation can reduce production effort, but errors in names, technical terms, speaker labels and non-speech sounds still need review for important content. A spoken answer should also have a readable equivalent; a visual highlight should include a textual description and location.
Accessibility should be designed into the interaction rather than added to the final media. Users need keyboard controls, adjustable playback, visible recording state, correction of transcripts and a way to choose text, speech or both without losing information.
- Provide synchronized captions and a separate transcript where appropriate.
- Describe important visual information rather than only transcribing speech.
- Do not require voice, sight or precise gestures for essential actions.
- Test with assistive technology and users, not only automatic conformance checks.
Every added modality creates new data and new instructions
Audio and video can reveal faces, voices, locations, bystanders, health cues, documents on a desk and conversations that were not intended for the service. A transcript is not a privacy-neutral substitute; it can make sensitive speech searchable and persistent. Derived embeddings and speaker labels also need retention and access rules.
Multimodal prompt injection expands the security problem. Instructions can be embedded in a document, screenshot, sign, subtitle or physical scene. OWASP specifically warns that hidden image instructions may alter model behavior. Recent research has demonstrated image-only and physical cross-modal attacks under experimental conditions.
The correct response is not to trust one detector. Treat media as untrusted content, separate system policy from extracted text, constrain tools, validate outputs and require approval for consequential actions. Face or voice resemblance should not become authentication merely because the model can describe it.
- Ask for consent at capture and explain what is retained.
- Minimize raw-media storage and separate it from derived records.
- Strip or isolate untrusted instructions found inside media.
- Keep identity proofing, authorization and transaction approval outside the multimodal model.
Long media and live sessions change the economics
Media processing adds upload time, storage, decoding, tokenization and specialist inference. Live sessions can repeatedly process accumulated context. Transcription, speech synthesis, video frames and tool calls may be billed separately or represented with different token rates.
A cost-aware product routes by task. A receipt can use a cropped image instead of a whole camera stream. A long meeting can be transcribed once, indexed and queried through selected segments. A video can receive a coarse scene pass before high-resolution analysis of a few moments.
Operations teams need modality-specific telemetry: dropped audio, frame rate, file conversion failures, token use, first-response latency, interruption rate, accessibility failures and deletion status. A single “request succeeded” metric cannot show why users stopped trusting the interface.
- Estimate cost from real media duration and resolution.
- Cache stable derived artifacts such as approved transcripts.
- Use progressive analysis instead of maximum resolution everywhere.
- Version preprocessing, encoders, models and output voices together.
What a benchmark worth believing should report
A performance number means little unless the workload, system configuration and quality bar are fixed. This is the minimum record a team should keep.
| Metric | How to measure it | Why it matters |
|---|---|---|
| Per-modality fidelity | Evaluate OCR, transcription, speaker attribution, object and document extraction on the real input distribution. | Cross-modal reasoning cannot repair systematic perception errors. |
| Cross-modal grounding | Require answers to connect text, audio and visual evidence and verify the referenced page, region, speaker or timestamp. | A model may answer from one modality while appearing to integrate several. |
| Temporal alignment | Score event order, audio-video synchronization and timestamp accuracy on short and long media. | Correct content attached to the wrong moment can reverse meaning. |
| Conflict handling | Present contradictory modalities and measure whether the system surfaces, resolves or asks about the conflict according to policy. | Silent averaging creates claims that no source supports. |
| Missing-modality robustness | Remove audio, images, captions or metadata and verify that confidence and behavior change appropriately. | Products often receive partial or damaged media. |
| Language and acoustic coverage | Test accents, code-switching, low-resource languages, noise, echo, overlap and non-speech sounds. | Average English transcription scores hide who the voice interface excludes. |
| Resolution and sampling sensitivity | Repeat tasks across image sizes, crops, frame rates and audio settings. | Preprocessing can determine the answer before model reasoning begins. |
| Live turn quality | Measure end-of-turn errors, interruption response, overlap, first audio latency and recovery after reconnect. | A fast model can still produce an unusable conversation. |
| Accessibility equivalence | Compare information available through text, speech, captions, descriptions and keyboard interaction. | Adding modalities should not make essential information dependent on one sense. |
| Calibration and abstention | Score confidence and refusal when media is blurry, incomplete, contradictory or outside the tested range. | An honest unknown is safer than a polished guess about missing evidence. |
| Multimodal injection resistance | Use visible, hidden and physical instructions in images, documents, audio and video while monitoring tool behavior. | Every parsed modality can carry adversarial instructions. |
| Privacy and retention | Verify consent, bystander handling, redaction, access control, deletion and derived-data retention. | Raw media and transcripts can expose more than the user intended to share. |
| Cost and latency per accepted task | Include upload, preprocessing, all modality tokens, tools, storage, review and retries. | Media cost is hidden when teams track only text generation. |
| Version regression | Canary changes to frame sampling, speech models, encoders, prompts and hosted model versions. | A multimodal update can improve one sense while quietly degrading another. |
Four sensible deployment patterns
Orchestrated specialist pipeline
- Where it fits
- Documents, meetings and regulated workflows that need inspectable OCR, transcription and deterministic validation.
- What you take on
- Components are replaceable and auditable, but handoffs add latency and alignment errors.
Unified hosted multimodal model
- Where it fits
- General assistants and real-time experiences where natural cross-modal conversation matters.
- What you take on
- Integration is fast and context may be richer, while internals, retention and version changes are less controllable.
On-device capture with cloud reasoning
- Where it fits
- Mobile, wearable and privacy-sensitive products that can redact, compress or detect locally before upload.
- What you take on
- Reduces raw-data exposure and bandwidth but introduces device fragmentation and split debugging.
High-assurance evidence workflow
- Where it fits
- Healthcare, finance, safety, legal and operational decisions where media influences consequential action.
- What you take on
- Preservation, provenance, specialist checks and human approval improve defensibility at higher cost and slower turnaround.
Where projects usually go wrong
The important video moment falls between samples
What you see: A brief event, code or gesture never reaches the model.
What to do: Use task-aware frame rates, scene detection and high-resolution reprocessing around suspected moments.
Tiny text changes during resizing
What you see: A label, decimal or identifier is read incorrectly while the scene description remains plausible.
What to do: Crop the source region, retain the original resolution and validate critical fields independently.
The right words are assigned to the wrong speaker
What you see: Meeting actions or consent are attributed to another participant.
What to do: Evaluate diarization, preserve timestamps and require review for consequential attribution.
Audio and video drift apart
What you see: A spoken instruction is connected to the wrong slide, object or action.
What to do: Use shared timestamps, monitor clock drift and test synchronization after transcoding.
One modality dominates conflicting evidence
What you see: The answer follows a caption or slide while ignoring contradictory audio or source data.
What to do: Define field-level authority and surface conflicts explicitly.
A transcript becomes the only record
What you see: Tone, overlap, non-speech events and correction evidence disappear.
What to do: Retain approved references and store raw media only under a defined need and retention policy.
A picture contains instructions for the model
What you see: Text in an image or document changes the workflow or triggers an unrelated tool.
What to do: Treat extracted content as untrusted, isolate policy and validate every action outside the model.
Face or voice is mistaken for authentication
What you see: A convincing recording or model judgment authorizes access or a transaction.
What to do: Use approved identity and transaction controls independent of media interpretation.
The system records more than the user realizes
What you see: Bystanders, background screens or private conversation enter storage and logs.
What to do: Use visible capture indicators, scoped consent, minimization, redaction and deletion controls.
The response is accessible in only one mode
What you see: A spoken answer has no text, or a highlighted visual region has no description.
What to do: Provide equivalent text, captions, descriptions and keyboard-operable controls.
A long live session becomes increasingly expensive
What you see: Latency and cost rise as prior audio and media remain in context.
What to do: Compress or summarize state deliberately, retain structured decisions and set session budgets.
A model update moves the weakest modality
What you see: Text quality improves while transcription, spatial reasoning or timing regresses.
What to do: Pin or canary versions and rerun a balanced multimodal acceptance set.
A checklist you can actually use
- Define the user task and which modalities are truly necessary.
- Show clear microphone, camera, screen-share and recording state.
- Obtain appropriate consent and plan for bystanders and shared spaces.
- Preserve the original file or stream reference when evidence matters.
- Record every resize, crop, sample rate, frame rate and transcription step.
- Choose specialist or unified architecture from measured requirements.
- Keep timestamps, page numbers, regions and speaker labels through the pipeline.
- Set field-level rules for which source is authoritative.
- Surface contradictions between modalities instead of hiding them.
- Escalate blurry, noisy or incomplete media rather than guessing.
- Test real devices, formats, accents, languages and environmental noise.
- Use tasks that require genuine cross-modal evidence in evaluation.
- Measure live interruption, turn detection and reconnect behavior.
- Provide captions, transcripts, descriptions and non-voice controls.
- Treat all media and extracted text as untrusted input.
- Keep tool permissions, identity proofing and transaction approval outside the model.
- Minimize raw-media retention and govern transcripts and embeddings too.
- Redact secrets and personal data before model or log exposure where possible.
- Budget image resolution, audio duration, video sampling and session history.
- Track cost and latency per accepted task, not per text response.
- Version preprocessing, encoders, models, prompts and output voices together.
- Canary updates across every modality and preserve rollback.
- Document known blind spots in the user interface and operating procedure.
Terms worth knowing
- Modality
- A form of input or output such as text, image, audio, video, depth or motion data.
- Multimodal model
- A model or system that processes or generates more than one modality.
- Omni model
- A broadly multimodal model designed for several input and output types, often including real-time audio.
- Modality encoder
- A component that converts media such as an image or audio waveform into representations usable by another model.
- Shared embedding space
- A representation where related items from different modalities are placed near one another for comparison or retrieval.
- Visual token
- A model input representing an image region, tile or compressed visual feature.
- Frame sampling
- Selecting a subset of video frames for processing instead of analyzing every frame.
- Diarization
- Separating and labeling who spoke when in an audio recording.
- Temporal alignment
- Connecting information from different streams to the correct time or event.
- Voice activity detection
- A method for estimating when speech begins and ends in a live audio stream.
- Cross-modal grounding
- Linking a claim or response to supporting evidence across two or more modalities.
- Media resolution
- The amount of visual detail allocated to an image or video frame during processing.
- Descriptive transcript
- A text record that includes relevant speech, non-speech audio and visual information.
- Multimodal prompt injection
- Adversarial instructions delivered through images, documents, audio, video or combinations of modalities.
- Progressive analysis
- Beginning with a lower-cost pass and selectively reprocessing important regions or time ranges at greater detail.
Primary references and technical starting points
These sources support the architecture, runtime, benchmark and security claims. Vendor capabilities can change, so the article records the distinction between established evidence, measured product behavior and editorial interpretation.
- 01 OpenAI API: Realtimeplatform.openai.com
- 02 OpenAI: Advancing voice intelligence with new API modelsopenai.com
- 03 OpenAI: GPT-4o System Cardopenai.com
- 04 Google: Gemini Live API overviewai.google.dev
- 05 Google: Gemini Live API capabilitiesai.google.dev
- 06 Google: Gemini image understandingai.google.dev
- 07 Google: Gemini audio understandingai.google.dev
- 08 Google: Gemini video understandingai.google.dev
- 09 Google: Gemini document understandingai.google.dev
- 10 Google: Multimodal token accountingai.google.dev
- 11 Qwen: Qwen2.5-Omni — See, Hear, Talk, Writeqwenlm.github.io
- 12 Qwen2.5-Omni Technical Reportarxiv.org
- 13 Meta AI: ImageBind — One Embedding Space to Bind Them Allai.meta.com
- 14 MMMU: A Massive Multi-discipline Multimodal Understanding Benchmarkarxiv.org
- 15 MMMU-Pro: A More Robust Multimodal Understanding Benchmarkarxiv.org
- 16 MMAU: Massive Multi-Task Audio Understanding and Reasoningarxiv.org
- 17 Video-MME: Comprehensive Video Understanding Evaluationarxiv.org
- 18 Video-MME-v2: Robust and Faithful Video Understandingarxiv.org
- 19 W3C WAI: Making Audio and Video Media Accessiblew3.org
- 20 W3C WAI: Captions and subtitlesw3.org
- 21 W3C WAI: Transcriptsw3.org
- 22 W3C WAI: Description of visual informationw3.org
- 23 NIST AI 600-1: Generative AI Profilenist.gov
- 24 OWASP LLM01:2025 Prompt Injectiongenai.owasp.org
- 25 Image-based Prompt Injection in Multimodal LLMsarxiv.org
- 26 Physical Prompt Injection Attacks on Vision-Language Modelsarxiv.org