Grayhat is an AI software engineering company advancing next-gen tech. Previously, we researched for 2 years and open-sourced deepdub. We aim to make a big impact in future tech (yes, big dreams!) and are figuring things out as we go. Most of our work is building products in Silicon Valley, and researching in next-gen technologies.
Today we're open-sourcing VidXP, a toolkit for indexing, understanding, and searching video collections using modern multimodal AI–after two years of research and development.
Videos contain an enormous amount of information, but finding the exact moment you need is still surprisingly difficult. VidXP automatically extracting transcripts, understanding scenes, generating embeddings, and making every frame searchable through natural language.
While technology like this has existed before (e.g. YouTube transcripts, Google Photos search, and Microsoft Video Indexer), and great strides have been made in research in video indexing and retrieval (like Google's VideoPrism, VideoCLIP, and InternVideo), the race towards private and decentralized AI motivated us to build tech like VidXP for the average consumer.
Whether you're working with research datasets, security footage, recorded meetings, educational content, podcasts, or personal archives, VidXP helps transform raw video into a searchable knowledge base.
Built for local-first AI workflows
Unlike many cloud-based video analysis platforms, VidXP is designed to run locally. Your videos remain under your control while leveraging state-of-the-art open models for speech recognition, semantic understanding, and retrieval.
The project can be used as:
- A Python library
- A command-line interface
- A REST API
- An MCP server for AI assistants
This makes it easy to integrate into existing applications or use directly from developer tools like Claude Desktop, Cursor, or any MCP-compatible client.
Spare the gibberish... What can VidXP do?
VidXP combines multiple AI capabilities into a single indexing pipeline.
What does this mean?
First, it processes your video (yes, that may take a while).
Then, you can start searching within those videos.
For example, you can simply ask:
"Find where a red car appears."
or
"Show me where the presenter explains vector databases."
VidXP returns the most relevant moments, as timestamps within that video.
(For nerds) How does VidXP do all the above? During processing, it will:
- Extract accurate transcripts
- Generate semantic embeddings
- Perform natural-language search across videos
- Retrieve precise timestamps for matching moments
- Index large video collections
- Serve search results through an API or CLI
Designed for...
Humans and AI
VidXP is available as a desktop app and an MCP Server for AI Agents (What is an MCP Server?).
Some people have used it to find certain events in their wedding videos, while others use it in their daily video editing workflow. Do let us know what use cases you come up with!
We're planning to release a Premiere Pro plugin based on VidXP. If you're interested, comment "Premiere Pro plz".
Developers
VidXP is intentionally modular.
You can embed it into your own applications, automate indexing pipelines, integrate it with retrieval systems, or expose your media archive to LLM-powered agents. Because it supports multiple interfaces, the same indexing engine can power local scripts, backend services, desktop tools, and AI workflows.
You can pick and choose the modules (we call them "collectors"), and can choose to plug your own SLM (Small Language Model) instead of the one we use.
Whether you're building media management software, research tooling, enterprise knowledge systems, or personal archives, VidXP provides the foundation for semantic video retrieval.
Standing on the shoulders of giants
VidXP wouldn't exist without the incredible open-source ecosystem that came before it. Rather than reinventing foundational AI models, we focused on building the glue that brings them together into a seamless developer experience.
Under the hood, VidXP builds upon state-of-the-art work from organizations including OpenAI, Anthropic, Google DeepMind, Microsoft, Meta AI, Alibaba, Nomic AI, BAAI, Hugging Face, and the countless researchers and open-source contributors who continue advancing machine learning.
Depending on your configuration, VidXP leverages technologies such as:
- Whisper and WhisperX for speech recognition and word-level alignment.
- PySceneDetect for automatic scene detection.
- OpenCV and FFmpeg for video processing.
- CLIP, OpenCLIP, and SigLIP for multimodal vision-language understanding.
- BGE and Nomic Embed for semantic embeddings.
- Sentence Transformers for embedding pipelines.
- Transformers for running modern multimodal models.
- FAISS and Chroma for vector search and retrieval.
Our goal has never been to replace these incredible projects. Instead, VidXP aims to make them easier to combine, deploy, and use together—whether from a Python library, CLI, REST API, or MCP server.
If VidXP saves you hours of engineering, it's because thousands of researchers, engineers, and open-source contributors spent years building the technologies it stands upon.
To everyone behind these models, libraries, datasets, and frameworks: thank you. ❤️
Available today
VidXP is available now, here:
grayhatdevelopers
(you can download the desktop app from our Releases page)
If you've ever wished you could search your videos as easily as searching documents, give it a try—we're excited to see what you build with it.