File Conversion 🔄

Most files are uploaded in a format that browsers cannot display, so the Hub asks Zamzar for a rendition — a second copy in a format that can be viewed, or that a downstream consumer such as the Mindset knowledge bank can read.

A rendition is its own row in the files table, with parent_id pointing at the original and file_path pointing at the converted object in S3. The original upload is kept exactly as it was.

The conversion round trip

Conversion is asynchronous and spans two services. The api/ service decides what needs converting and records the result; the pdfKue/ service talks to Zamzar and moves bytes. They communicate over three Bull queues on Redis.

• Queue names are configurable (CONVERSION_REQUEST_QUEUE, CONVERSION_DONE_QUEUE) and default to convert and converted — see api/config/conversion.js. Each job is retried up to five times.
• pdfKue polls Zamzar rather than receiving a callback, via its own internal pending queues — one for video, one for everything else.
• Everything runs against the private S3 bucket, and renditions are served through signed CDN URLs rather than directly.

What gets converted

convertFiles() (api/modules/conversion/index.js) is called for every uploaded file from postProcessFiles() (api/modules/files/processors/assets.js). It decides between four outcomes:

An unknown file type is still usable: the asset is flagged download: REQUIRED so the Hub offers it as a download rather than trying to render it in the asset viewer.

Conversion targets

Target formats come from api/modules/supportedFileTypes.js.

Type	Converted to	Notes
Documents	pdf	Text is extracted from the PDF and stored on the file row
Images	png
Audio	mp3
Video	mp4	Only when the format cannot be read by the AI knowledge bank — see below
Archives	—	zip files are handled by the unzipper, not by conversion
Subtitles	—	vtt is served as uploaded

A file whose extension already matches its type's target format is never sent for conversion — the mapping in supportedFileTypes.js omits convertTo in that case.

File state

files tracks processing in two places, written by different pipelines, and knowing which is which matters.

Column	Written by	Read by
state	Brightcove for video, the conversion pipeline for everything else	Asset viewer, playlist pages, content cards, post-processing emails
video_rendition_state	The MP4 rendition pipeline only	The video rendition sweep
video_rendition_requested_time	Stamped whenever a rendition is requested	The lease that makes a lost conversion retryable
video_rendition_attempts	Incremented on each request	The cap that stops a video being asked about forever

state moves through draft → processing → ready or invalid (api/modules/fileUploadStatus.js).

The rendition columns are video-only. video_rendition_state is NULL until a rendition is asked for, and then moves to processing → ready, invalid or too_large. NULL means no MP4 rendition applies, which covers every document, image and thumbnail, and every replica of a file another region owns.

Why video needs its own column

Video is played through Brightcove, whose ingest only picks up a file while its state is draft. Conversion and ingest run against the same file at the same time, so conversion must leave state alone — a video already marked ready is skipped by the ingest and never becomes playable. The separate column keeps each pipeline blind to the other's progress.

Video renditions

Videos are converted for one reason: the Mindset knowledge bank can only read video/mp4, so a video uploaded as .mov, .avi, .wmv, .mkv, .webm, .flv, .mpg, .m4v or .ts is invisible to it. The rendition gives it something to ingest.

Playback is unaffected either way. Video plays through Brightcove, which transcodes independently and ignores the rendition entirely.

api/modules/conversion/videoRenditions.js handles this path:

• .mp4 is skipped — the knowledge bank reads it as uploaded, so there is nothing to gain and a Zamzar credit to lose. Eligibility is read from the file name rather than mime_type, which the browser supplies and gets wrong often enough ('', application/octet-stream) to buy a billed mp4-to-mp4 conversion.
• New uploads convert immediately, in the same postProcessFiles() call that triggers the Brightcove ingest.
• Videos already in the database are picked up by queueMissingVideoRenditions(), called once for every Mindset Hub from the Mindset sync and capped per run.
• The region that owns the file converts it. A rendition reaches every other region through ordinary content sharing, so converting a replica would pay Zamzar twice for one file and write a rendition its owner never sees. A Hub shown content another region owns publishes a REQUEST_VIDEO_RENDITION message on the existing playlistSync topic, and that region does the work. Without it a video distributed to a Mindset Hub whose owning Hub has no Mindset sync of its own would never be converted by anybody.

Recovering from failures

Conversion crosses a queue, a third-party API and S3, so requests do get lost. Four rules keep a video from being stranded in a state it can never leave:

• A request that cannot be queued leaves the state untouched, so the next sync tries again. Marking it failed would permanently strand a video over what is usually a brief Redis outage.
• Asking is bounded, however the last attempt ended. A request still processing after 24 hours is taken as lost — pdfKue publishes nothing when a worker exhausts its retries, and a real conversion finishes in minutes. A request that came back invalid is tried again on the same interval, because a Zamzar outage or a dropped download says nothing about the video. video_rendition_attempts caps the total at VIDEO_RENDITION_MAX_ATTEMPTS, so a video that genuinely cannot be converted stops consuming credits. video_rendition_requested_time is what makes the interval measurable; updated_time cannot, since any unrelated write to the row moves it.
• A file beyond the Zamzar plan's size limit is marked too_large, not invalid, so "never attempted" stays distinguishable from "attempted and failed" and those videos remain findable. The check is inert unless ZAMZAR_MAX_SOURCE_BYTES is set.
• A completed rendition is never downgraded. A retried request can report failure after the original already succeeded, and a rendition that exists on S3 must not be marked invalid because of it. For the same reason, a repeated success does not record the rendition twice.

pdfKue

pdfKue/ is a standalone worker; its own README covers running it. Two behaviours are worth knowing from the Hub side:

• PDFs are buffered, everything else streams. Text extraction needs random access to the whole document, so PDF targets are read into memory. Any other target — a converted video especially — is streamed straight into the S3 upload, which would otherwise be too large to hold.
• Video waits on its own queue. A conversion holds a worker for as long as Zamzar takes plus the whole transfer, so video jobs go to pendingVideoConversion and everything else stays on pendingConversion, each with its own concurrency. Sharing one queue lets a batch of videos delay every document behind it for hours.
• Downloads abort on inactivity rather than elapsed time. A large video moving steadily is healthy however long it takes, and a socket that has stopped delivering is not worth waiting on. The same abort handle cancels the download when the S3 upload fails, instead of leaving the connection open until a timer expires.
• Zamzar bills per credit, and the cost of each job is logged as creditCost alongside targetFormat on the Conversion job finished line, so spend per format is a log query.

Adding a new file type

To support a new file extension, add it to the list of supported types in all three files:

• api/modules/supportedFileTypes.js
• frontend/src/common/utils/asset/setAssetTypes.js
• frontend/src/common/utils/file/validFileTypes.js

Then check that Zamzar itself supports converting from that format to the type's target, and add it to the supported file types list.