Plugin-Extensible, Timestamp-Aware Search Results in Jellyfin #18561
murphy52
started this conversation in
Feature Proposals
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
TL;DR:
Extend Jellyfin's plugin search API to support timestamped results within videos, allowing search plugins to return specific chapters, songs, scenes, or dialogue that users can jump directly to. Results would appear in native search with thumbnails using existing trickplay, chapter images, or primary artwork.
Summary
I'd like Jellyfin's search experience to support results that identify a specific moment within a media item, rather than only the media item as a whole. A search provider could return a labeled match with a timestamp, such as a song performed in a concert, a scene in a film, a topic in a lecture, or a phrase in a recorded interview. Selecting the result would start playback of the existing item at that time.
The goal is not to add any particular indexer, transcription engine, or metadata format to Jellyfin core. It is to create a small, reusable contract that plugin authors can use to provide timestamped results and that Jellyfin clients can render consistently. Core would handle playback navigation and standard image resolution.
This is intended to extend, not replace, the plugin search-provider facilities that have been introduced in recent versions of Jellyfin. The precise interface and version-specific integration details should be validated with maintainers before an implementation is proposed.
The problem
Some collections have useful information below the media-item level. A single concert video can contain 20 songs; a long lecture can contain 15 topics; a documentary can have named segments; a home video can contain several events. That information may already exist as chapters, external annotations, transcripts, or indexes maintained by plugins.
When built-in search only returns the parent video, the user must open that item and navigate manually to the desired moment. A plugin may know the exact timestamp but has no universal way to represent that result inside Jellyfin's native search UI, particularly across different TV clients.
A separate website or phone-based search UI is not an ideal solution. The desired experience is search and direct playback within the Jellyfin client the viewer is already using.
Proposed behavior
This should be additive. Existing search results and clients should continue working without requiring timestamp-aware providers.
Illustrative search result contract
The structure below is conceptual, not a claim about Jellyfin's present API or a proposed final public type name.
{ "resultId": "provider-id:stable-match-id", "providerId": "example-provider", "mediaItemId": "existing-jellyfin-item-id", "label": "Interview segment: early recording sessions", "startTicks": 18200000000, "endTicks": 24000000000, "relevanceScore": 0.92, "description": "Discussion of the first album", "matchType": "segment", "thumbnailOverride": null }A practical contract should preserve whatever time representation, user scoping, cancellation behavior, and score conventions Jellyfin already uses.
resultIdmust distinguish two matches within a single media item;mediaItemIdalone is insufficient. A stable provider-qualified result ID also helps with UI state, caching, and deduplication.Core minimum: media item ID, start timestamp, display label, and enough identity to distinguish and select a result. Everything else should be evaluated for necessity before adding it to a public plugin API.
Thumbnail handling
Timestamped matches should have a useful preview without forcing plugins to implement image extraction. I suggest the following fallback order:
The reason to prefer trickplay over chapter images is that a timestamped search result need not coincide with a chapter boundary. The image closest to the actual match time is generally the most relevant preview.
Chapter images and trickplay are different existing systems within Jellyfin. Chapter images are individually associated with chapters; trickplay typically uses tiled sprite sheets. Therefore an arbitrary timestamp cannot simply be assumed to correspond to an individual ready-to-serve JPEG. The thumbnail-resolution abstraction should hide the underlying source from search providers and, ideally, from clients.
One possible solution is a server-side endpoint that accepts an item ID and timestamp and returns or redirects to a renderable thumbnail, with appropriate caching. However, the costs of decoding or cropping a trickplay tile on demand should be considered carefully. A client-side helper or a shared representation of sprite coordinates might be preferable in some implementations. That detail is a topic for maintainer feedback, not a prerequisite of the data-model proposal.
The absence of trickplay or chapter images must not block search results. Thumbnail resolution should be opportunistic, efficient, and able to fall back to existing primary artwork.
Search and ranking
Timestamped results should be returned alongside, but visually distinguished from, ordinary media items. A dedicated group allows users to understand why multiple search hits point into the same video. It also prevents dozens of matches in one long recording from drowning out normal title results.
Potential policies to discuss:
There should be no requirement for a core full-text engine or AI model. Providers could use chapter names, transcripts, human-created indexes, or external metadata.
Playback behavior
Selecting a timestamped result should pass the parent media item and requested playback position into the client's ordinary playback flow. It should not create a virtual media item or split files. Resume-state behavior and playback bookkeeping should continue to apply to the actual parent video.
Important details:
Server and plugin responsibilities
Providers: decide what is indexed, maintain their own index, generate matches, return identifiers/timestamps/labels/scores, and optionally specify images or descriptions. Providers should not need to implement a full playback UI or thumbnail extraction.
Server: aggregate and bound plugin results; preserve provider-qualified match identity; validate visibility and authorization; expose results in a stable API; offer a compatible way to obtain preview imagery; and avoid regressions to existing search results.
Clients: present timestamped matches distinctly; show the associated title/time/thumbnail; and launch normal playback at the supplied position. There should be a documented feature-detection or capability mechanism so older clients can ignore new result types cleanly.
This intentionally keeps the design useful for many independent plugin types rather than coupling core to chapter metadata or a particular music use case.
Security, privacy, and operational considerations
Possible phased rollout
Phase 1: Agree on the contract. Define the minimum provider result shape, permissions model, and backward-compatible search API behavior. Use a minimal demonstration provider that returns multiple timestamps from the same media item.
Phase 2: Implement one end-to-end client path. Render timestamped results as a distinct group and start playback at the selected position, initially using ordinary parent artwork if necessary. This is the critical functional slice.
Phase 3: Centralize thumbnail resolution. Add optional overrides and the fallback sequence: trickplay at timestamp, then relevant chapter image, then primary artwork. Decide whether server-side thumbnail extraction is appropriate or whether a shared client-friendly representation is preferable.
Phase 4: Expand client support and polish. Make the behavior consistent in web, Android TV, and other maintained Jellyfin clients, accounting for differing release cycles. Optimize ranking, caching, accessibility, and performance based on real collections.
This order keeps thumbnail sophistication from blocking a usable first implementation.
Acceptance criteria for an initial implementation
Questions for Jellyfin maintainers
Why this belongs in core
The underlying metadata and indexing can remain entirely in plugins. But the ability to display a result that means “play this video, at this position” is a cross-client capability. If every plugin builds a separate interface, users lose consistency and television support becomes difficult. A compact core contract and standard client behavior would allow many kinds of plugins to offer the same native experience without making Jellyfin core responsible for understanding their content.
I'd welcome feedback on whether the data model should be narrower, where thumbnail resolution should live, and which minimal vertical slice would be most useful to demonstrate first.
All reactions