Every media pipeline eventually produces a file where the mouth moves and the words arrive a beat later. The usual reaction is to re-encode and hope. The useful reaction is to understand PTS and DTS — the two timestamps every compressed packet carries, why they are different numbers, and why the units they are expressed in are not seconds. Almost everything that goes wrong with audio/video sync is a timestamp problem wearing a codec’s clothes, and almost nobody explains it outside developer documentation.
This is the explanation. It is written for someone building or debugging a media pipeline, not for someone looking for a magic flag to paste. By the end you should be able to look at a desynchronised file, read its timestamps, and say which of four distinct failure modes you are actually looking at.
Version note. Command-line behaviour in this area has changed across releases — the frame-rate-mode option was respelled, and audio-sync handling migrated into the resampler. Everything below describes the FFmpeg 8.1 “Hoare” line (ffmpeg.org’s download page lists 8.1 “Hoare”, released 2026-03-16, as the current stable release from that branch, checked 2026-08-14). Where a flag’s spelling or default matters to you, confirm it against your own build with ffmpeg -h full rather than trusting any article — including this one.
One more piece of vocabulary before we start, because the rest of the article depends on it. A container (MP4, Matroska/MKV, WebM, MPEG-TS) is the envelope: it holds one or more encoded streams plus their metadata, their timestamps, and usually a seek index. A codec (H.264, HEVC, AV1, AAC, Opus) is the compression scheme applied to the data inside. Muxing is the act of interleaving encoded streams into a container; demuxing is pulling them back out. Timestamps live in the container layer, which is why sync problems survive a change of codec and why they can be introduced by a change of container.
Two Timestamps, One Stream: What PTS and DTS Actually Mean
FFmpeg’s own data structures state the distinction more precisely than any tutorial. In the AVPacket struct — the type that carries compressed data between a demuxer and a decoder, or between an encoder and a muxer — the Doxygen reference defines:
- PTS (Presentation Time Stamp) — “the time at which the decompressed packet will be presented to the user,” expressed in
AVStream->time_baseunits. - DTS (Decoding Time Stamp) — “the time at which the packet is decompressed.”
And it states the invariant that binds them: PTS must be greater than or equal to DTS, because a frame cannot be shown before it has been decoded.
That is the whole model. One number answers when should the decoder start work on this? The other answers when should the viewer see the result? For audio, and for video with no forward references, the two numbers are the same and nobody ever thinks about them. For most real video, they are not.
Note also which structure carries which. AVPacket — compressed, pre-decode — carries both PTS and DTS. AVFrame, the decoded raw picture, carries only a presentation timestamp: “time when frame should be shown to user.” By the time you hold a frame, reordering has already happened and decode order has ceased to be meaningful. That asymmetry is a good mental checkpoint — if you are looking at DTS, you are still in packet-land.
Why Decode Order Isn’t Display Order
Video codecs compress by predicting. An I-frame is self-contained — a complete picture coded without reference to any other frame. A P-frame is predicted from an earlier decoded frame, so unchanged regions never get re-sent. A B-frame is predicted by combining regions from two previously decoded frames — one earlier in the timeline and one later. The repeating arrangement of these between keyframes is a GOP (Group of Pictures).
That last one is the root of the entire problem. A B-frame sits, in display order, between the two frames it references — but it cannot be decoded until both of those frames are already decoded, including the one that comes after it on screen. So the encoder must write the future frame into the bitstream first, and the decoder must hold it aside while it decodes the B-frames that depend on it. Coded order and display order genuinely diverge, and the container has to carry both orderings explicitly. That is why DTS exists at all.
Here is a worked example: a seven-frame IBBPBBP pattern at 30 fps, with a two-frame reorder delay, in a stream whose time base is 1/30 of a second — so one tick equals one frame.
| Decode slot | Frame (display position) | DTS (ticks) | PTS (ticks) | Depends on |
|---|---|---|---|---|
| 0 | I₀ | 0 | 2 | — (self-contained) |
| 1 | P₃ | 1 | 5 | I₀ |
| 2 | B₁ | 2 | 3 | I₀ and P₃ |
| 3 | B₂ | 3 | 4 | I₀ and P₃ |
| 4 | P₆ | 4 | 8 | P₃ |
| 5 | B₄ | 5 | 6 | P₃ and P₆ |
| 6 | B₅ | 6 | 7 | P₃ and P₆ |
Read the two orderings out of the table:
- Decode order (DTS ascending): I₀ → P₃ → B₁ → B₂ → P₆ → B₄ → B₅
- Display order (PTS ascending): I₀ → B₁ → B₂ → P₃ → B₄ → B₅ → P₆
Three things in that table are worth staring at.
DTS is strictly monotonic; PTS is not. Decode order is the order bytes physically appear in the stream, so DTS only ever counts up. PTS jumps forward and back. Any tool that assumes timestamps increase monotonically will mangle this — and a surprising amount of homegrown pipeline code makes exactly that assumption.
There is a startup delay, and it is real. Presentation begins at tick 2, not tick 0, because the decoder needs two frames in hand before it can emit the first one in display order. A deeper reference structure means a deeper reorder buffer and a longer delay. This is why low-latency encoding configurations often disable B-frames outright: the compression is worse, but the pipeline stops owing the viewer a multi-frame debt.
The offsets are per-frame, not constant. PTS − DTS is +2 for I₀, +4 for P₃, +1 for each B-frame. There is no single “sync offset” to add. Some muxers instead keep decode times starting at zero and let early DTS values go negative, then apply a global shift at mux time so nothing on disk is below zero. Both representations are legitimate; they describe the same stream. If you compare timestamps between two tools and see a constant difference across the whole file, suspect a normalising shift like that before you suspect corruption.
An open GOP — where boundary B-frames may reference across a keyframe into the neighbouring GOP — buys a little compression and complicates this further, because the reorder window straddles what looked like a clean cut point. A closed GOP is self-contained and safe to cut at. If you are building anything that splices, prefer closed GOPs and stop fighting the reorder buffer.
Time Bases: Why a Timestamp Isn’t a Number of Seconds
The second half of the difficulty is that PTS and DTS are integers in a stream-specific unit, not durations. FFmpeg’s AVStream documentation defines time_base as “the fundamental unit of time (in seconds) in terms of which frame timestamps are represented” — a rational number, a numerator over a denominator, not a fixed clock.
A raw PTS of 90000 means nothing on its own. In a stream with a time base of 1/90000 it is one second. In a stream with a time base of 1/1000 it is ninety seconds. Every timestamp is meaningless without the time base of the stream it came from, and each stream in a container carries its own.
Rationals rather than floating-point seconds is a deliberate, correct decision. Video frame durations are frequently not representable in binary floating point — NTSC-derived 29.97 fps is exactly 30000/1001, and 23.976 is 24000/1001. Store those as doubles and accumulate them over an hour and you will drift. Store them as exact rationals and the arithmetic stays exact until something forces a conversion.
Something usually forces a conversion. Here is what different layers actually use:
| Layer | Typical time base | One tick equals | Notes |
|---|---|---|---|
| MPEG-TS / MPEG program stream | 1/90000 | ≈ 11.1 µs | PTS in MPEG systems streams has “a resolution of 90kHz” — the classic broadcast clock |
| Matroska / WebM | 1/1000 by default | 1 ms | Matroska’s TimestampScale defaults to 1,000,000 ns, so “one Segment Tick represents one millisecond” |
| MP4 / ISOBMFF | Per-track timescale (commonly 90000, 15360, 12800, or the sample rate for audio) | varies | Chosen by the muxer; audio tracks very often use the sample rate itself |
| Audio streams generally | 1/sample_rate | one audio sample | 1/48000 for 48 kHz — the finest unit that means anything for that stream |
| A CFR video stream | 1/frame_rate | one frame | Convenient, but only valid while the rate really is constant |
Now the consequence. Matroska’s default tick is one millisecond. A 48 kHz audio sample is about 20.8 microseconds. Muxing that audio into MKV means every timestamp gets rounded to the nearest millisecond — up to roughly 24 samples of quantisation on each one. For a single packet that is inaudible. For a pipeline that repeatedly converts timestamps between bases, rounds, and then derives the next timestamp from the rounded value, the errors compound instead of cancelling.
That is the difference between a rounding error and a drift: a rounding error is bounded and independent per packet; a drift is what you get when rounded values feed the next calculation. Timestamps should always be rescaled from the original value, never accumulated from the previous rescaled one. FFmpeg’s internal rescaling helpers exist precisely to do this correctly, with explicit rounding control; code that re-derives timestamps by adding a nominal frame duration each time is the single most common way to build drift into a pipeline by hand.
Two Streams, Two Clocks: The Part Nobody Designed
Here is the structural fact that makes audio/video sync a genuinely hard problem rather than a solved one: audio and video are separate elementary streams with independent timestamp sequences, in independent units, and there is no such thing as “frame N” that means the same thing to both.
Nothing in the file format ties video frame 300 to a specific audio sample. All they share is a nominal common origin and the presumption that both sets of timestamps refer to the same underlying timeline. Sync is not stored; it is reconstructed at playback time by a player that maintains a master clock (usually the audio clock, because audio glitches are far more noticeable than a repeated or dropped video frame) and schedules video frames against it.
When those streams were produced by genuinely different hardware, the presumption of a shared timeline is an approximation. A camera’s sensor clock and its audio codec’s sample clock are separate crystal oscillators. A capture card and a USB microphone certainly are. If the audio device is running at a true 47,999.5 Hz while the container declares 48,000 Hz, the file has a real, constant, sub-part-per-thousand error that grows linearly — roughly a third of a second by the end of a nine-hour recording, and far more visible than that in a live stream that never ends.
This is why the tolerance question matters and why it has a standards answer rather than a folk answer: ITU-R Recommendation BT.1359-1, “Relative timing of sound and vision for broadcasting” (approved November 1998, still in force), is the reference broadcasters use for how much offset is acceptable. The perceptual window is asymmetric — sound arriving after the picture is treated differently from sound arriving before it, which matches ordinary physical experience, since in the real world sound always arrives late. A pipeline that treats sync as a symmetric ± tolerance is quietly using the wrong model.
Constant Frame Rate vs Variable Frame Rate
A second structural split sits underneath all of this. In a constant frame rate (CFR) stream, every frame occupies exactly the same duration and PTS values advance by a fixed step. In a variable frame rate (VFR) stream, they do not: frame durations vary, and the timestamps are the only record of when anything was actually meant to appear.
| Constant frame rate (CFR) | Variable frame rate (VFR) | |
|---|---|---|
| Frame duration | Fixed | Varies per frame |
| Where PTS comes from | Derivable: frame index × time base | Only from the recorded timestamp |
| Typical sources | Broadcast, most cameras, most encoders’ output | Screen recorders, phone cameras (adaptive rates in low light), game capture, WebRTC |
| Editing behaviour | Predictable | Timeline-based editors often conform it to CFR on import |
| Failure mode | Wrong declared rate → whole file plays fast or slow | Timestamps dropped or ignored → progressive drift |
| Sync risk | Low, and the error is constant | High, and the error accumulates |
VFR exists for good reasons. A screen recorder has nothing to encode while the screen is static, so emitting frames only on change is a real saving. A phone camera lengthens exposure in dim light, which lowers the capture rate. Neither is a defect.
VFR becomes a problem the moment something downstream treats it as CFR. Take a screen recording whose frames arrive anywhere between 5 and 60 fps, hand it to a tool that reads only the average frame rate and re-derives timestamps by counting frames, and the output will slowly walk away from its own audio. The video isn’t corrupted; the timestamps that carried the truth were thrown away and replaced with an assumption.
This is also why ffprobe reports more than one frame-rate field. r_frame_rate is the base rate the container declares — the lowest rate at which all timestamps can be represented as integers — while avg_frame_rate is total frames over total duration. On CFR content they agree. On VFR content they legitimately disagree, and the disagreement is the diagnostic, not an error.
What Actually Causes Drift
Four distinct mechanisms produce “the audio is out of sync,” and they need four different responses. Fixing the wrong one is why people end up re-encoding the same file five times.
| Cause | What is physically happening | Symptom you observe | What actually addresses it |
|---|---|---|---|
| Clock mismatch between capture devices | Two independent oscillators; declared sample rate ≠ true sample rate | Offset grows linearly from ~0; worst at the end of a long recording | Resample audio against the video clock (aresample with a compensation threshold); at capture time, genlock or share a clock source |
| Dropped or duplicated frames at capture | Encoder or disk couldn’t keep up; frames never reached the muxer | A step change in offset at the moment of the drop; often a burst under load | Fix the capture bottleneck; if timestamps survived, re-time to them rather than to a frame count |
| Container timestamp precision | Rescaling between time bases with rounding, compounded across packets | Slow, small, steady creep — often only visible over tens of minutes | Rescale from original values, not from previously rescaled ones; prefer a container with a fine enough tick for the content |
| Remux between containers with different time bases | Timestamps quantised into the destination’s units; edit lists, start offsets, or initial delays not carried over | Constant offset appearing immediately after a container change, or a fixed head-of-file shift | Inspect timestamps before and after; apply an explicit offset rather than re-encoding blindly |
| VFR treated as CFR | Real per-frame timestamps discarded and replaced by index × nominal duration | Drift that tracks how much the true rate varies | Keep the timestamps (-fps_mode passthrough) or deliberately conform to CFR (-fps_mode cfr) with eyes open |
The shape of the error tells you the cause. Linear growth from zero is a clock-rate mismatch. A step is a drop or a splice. A constant offset present from the first frame is a container/pipeline offset, not drift at all. Creep that only shows up over a long file is precision. Measure the offset at three points — start, middle, end — before touching a flag, because that alone narrows five candidates to one.
Network conditions add a sixth mechanism for live pipelines. Variable delivery latency doesn’t change the timestamps in the stream, but it changes when packets arrive relative to the playback clock, and a player with an undersized buffer will make visible decisions about it — which is why jitter and packet loss show up as sync artefacts even when the source file is perfect, and why available bandwidth is a sync variable in streaming and not only a quality one.
Reading Timestamps: ffprobe as the Diagnostic
Before changing anything, look at the actual numbers. ffprobe — one of the three CLI tools built on the FFmpeg libraries, alongside ffmpeg and ffplay — exists to “gather information from multimedia streams and print it in human- and machine-readable fashion.”
Start with the stream layer, because the time bases are the context for everything else:
ffprobe -v error \ -show_entries stream=index,codec_type,codec_name,time_base,r_frame_rate,avg_frame_rate,start_time,duration \ -of json input.mp4Read four things out of that. Do the video and audio start_time values agree? Do r_frame_rate and avg_frame_rate agree (CFR) or not (VFR)? What is each stream’s time_base, and is it fine enough for the content? Do the declared durations of the two streams match?
Then go down to packets, where PTS and DTS live:
ffprobe -v error -select_streams v:0 \ -show_entries packet=pts_time,dts_time,duration_time,flags \ -of csv=p=0 input.mp4 | head -40pts_time and dts_time are the raw integer timestamps already multiplied by the stream’s time base and printed in seconds, which is what you want for reading by eye. On a stream with B-frames you will see pts_time values that are not monotonic while dts_time climbs steadily — the reorder from the table above, in the wild. The flags column marks keyframes, which is where seeking and cutting can land cleanly.
Run the same query against -select_streams a:0 and compare the two streams’ first timestamps. A difference at packet zero is a constant offset. A difference that is small at the start and large at the end is drift. That single comparison is the highest-value diagnostic in this entire article, and it costs one command.
Two things worth knowing about ffprobe output before you over-interpret it. Container-level and codec-level facts are separate description layers and can legitimately disagree — a declared container duration and one computed by walking packet timestamps will diverge on files with edit lists or an incomplete index. And a file with no index at all (plain MPEG-TS carries no seek table) will give you approximate answers to structural questions by design.
The Flags That Move Timestamps
Once you know which failure mode you have, a small number of options address each one. All of the following describes the FFmpeg 8.1 line; verify against ffmpeg -h full on your build.
-fps_mode — what to do with video frame timing
This is the option that decides whether the video timeline is preserved, regenerated, or forced constant. In the current sources its help string reads “set framerate mode for matching video streams”, and its parser accepts four values: passthrough, cfr, vfr, and auto.
passthrough keeps each frame’s timestamps as they arrived. cfr produces a constant-rate output, duplicating or dropping frames to hit the target rate exactly. vfr allows frames through with their own timing but avoids emitting duplicates. auto lets the muxer decide.
Two practical notes. -fps_mode is a per-stream option — -fps_mode:v:0 passthrough — where the older global -vsync spelling it superseded was not. And this exact area is why the version pin at the top of this article exists: the spelling changed, and articles written against the old one are still the majority of what a search returns. Check the help output, not the blog post.
aresample — what to do with audio that doesn’t match its timestamps
Audio sync compensation lives in the resampler, and the resampler options documentation describes it precisely. async is “simple 1 parameter audio sync to timestamps using stretching, squeezing, filling and trimming. Setting this to 1 will enable filling and trimming, larger values represent the maximum amount in samples that the data may be stretched or squeezed for each second. Default value is 0, thus no compensation is applied to make the samples match the audio timestamps.”
So -af aresample=async=1 means fill gaps and trim overlaps so the samples match the timestamps — the right tool for a clock-mismatch drift, and the wrong tool for a constant offset. The finer controls are worth knowing:
min_comp— “set the minimum difference between timestamps and audio data (in seconds) to trigger stretching/squeezing/filling or trimming.” Default: compensation disabled (FLT_MAX).min_hard_comp— “a threshold to select between hard (trim/fill) and soft (squeeze/stretch) compensation.” Default0.1.max_soft_comp— “maximum factor by which data is stretched/squeezed.” Default0.first_pts— “assume the first pts should be this value… This allows for padding/trimming at the start of stream.”
The older top-level -async option expressed the same idea with less control; the resampler is where the documented, tunable version lives.
-itsoffset — deliberately shifting one input
For a constant offset, don’t stretch anything — move one input. -itsoffset is documented in FFmpeg’s own manual source as, exactly, “Set the input time offset.” It is an input option, so it binds to the -i that follows it:
ffmpeg -i video.mp4 -itsoffset 0.250 -i audio.wav \ -map 0:v -map 1:a -c copy output.mkvThat pushes the second input 250 ms later. Positional binding matters here more than almost anywhere else in the tool: an option placed before the wrong -i silently applies to the wrong input, because FFmpeg’s grammar scopes options to the next file specified and resets them between files. There is also -copyts, which tells FFmpeg to preserve input timestamps rather than normalising them — useful when you are trying to observe a problem rather than paper over it, and a good habit when producing a diagnostic copy.
Choose by failure mode, not by convenience: constant offset → -itsoffset; growing offset from a clock mismatch → aresample=async=...; timeline regenerated or destroyed → -fps_mode. Applying the drift tool to a constant offset produces a file that is wrong in a new and harder-to-describe way.
Why Remuxing Doesn’t Fix Bad Timestamps
The most common piece of advice in this entire subject is “just remux it,” and it is the piece most often misunderstood.
-c copy is documented plainly: “the stream is not to be re-encoded.” Compressed packets pass from the demuxer straight to the muxer without being opened. Decode, filter, and encode are skipped entirely, which is why remuxing a multi-gigabyte file to a different container takes about a second while transcoding it takes minutes and costs quality.
But the packets are not the whole file. Timestamps are rewritten on every remux, because the destination container has its own time base and its own rules. That is not a bug; it is what muxing is. Which means:
Remuxing preserves timestamps when the source timeline is sane, the destination’s time base is fine enough to represent it, and no start offsets or edit lists get lost in translation. This is the normal case, and the resulting file is genuinely bit-identical in its media data.
Remuxing reveals a problem far more often than it creates one. A file with an inconsistent timeline may play fine in one player — because that player is tolerant, resynchronising quietly against its audio clock — and expose itself immediately in another, or after a container change that discards the compensating metadata the first player was leaning on. The classic version of this is a source with a non-zero start offset or an MP4 edit list: the original player honoured it, the new container has no equivalent, and the offset that was being silently corrected is suddenly visible.
So the honest rule is: remuxing does not fix bad timestamps — it usually exposes them. If a file goes out of sync when you change its container, the timestamps were already wrong; the previous container was compensating. That is good news diagnostically, because now you can measure the problem instead of hoping the next player is also forgiving.
The corollary matters just as much. Because stream copy cannot re-time individual frames, -c copy is not a repair mechanism. It can shift a whole stream (-itsoffset) and it can move packets into a container with a better time base, but it cannot resample audio, cannot rebuild a destroyed VFR timeline, and cannot invent timestamps that were never recorded. Those operations require decoding. Knowing which class of fix your problem needs — before you type a command — is most of the skill.
❓ Frequently Asked Questions
Why does my video have PTS values that go backwards? It doesn’t — the stream is in decode order, and PTS is the display order, so reading packets sequentially shows PTS jumping back and forth while DTS climbs steadily. That is B-frame reordering working correctly. It becomes a real problem only when a tool in your pipeline assumes timestamps are monotonic and either errors out or silently re-sorts them.
Is PTS ever less than DTS? No. FFmpeg’s own AVPacket documentation states PTS must be greater than or equal to DTS, because presentation cannot precede decoding. If a tool reports otherwise, you are looking at a malformed stream, a value being read in the wrong time base, or a display bug — not a legitimate state.
Why can’t I just store timestamps in seconds as floating-point numbers? Because common frame rates aren’t exactly representable in binary floating point. 29.97 fps is exactly 30000/1001; store it as a double and accumulate frame durations across a long file and the error compounds. A rational time base keeps the arithmetic exact until you deliberately convert, and puts the conversion — and its rounding — somewhere you can see it.
My audio drifts out by about a second over an hour. Is that a codec problem? Almost certainly not. An offset growing linearly from roughly zero is the signature of a clock mismatch — the true sample rate of the capture device differs slightly from the declared one. Re-encoding without addressing the timestamps will faithfully reproduce the drift. Resampling audio against the video clock is the fix; aresample‘s async parameter is the documented control.
Does -c copy change my file’s timestamps? It leaves the compressed media data untouched, but timestamps are rewritten during muxing because the destination container has its own time base. Usually that’s lossless in any way you’d notice. When it isn’t — a coarser tick, a lost edit list, a dropped start offset — the difference shows up as sync error that was previously being hidden.
Should I use CFR or VFR for my pipeline output? If the output feeds an editor, a broadcast chain, or anything that assumes fixed frame timing, produce CFR and accept the duplicated or dropped frames. If it’s an archive of a variable-rate source such as a screen recording, keep the original timing with -fps_mode passthrough — conforming it to CFR is lossy in a way that cannot be undone later. The mistake is doing either one accidentally.
Further Reading
- FFmpeg —
ffmpegdocumentation — the option reference, including the positional-scoping rule that makes-itsoffsetbind to the input that follows it. - FFmpeg —
ffprobedocumentation —-show_entries,-select_streams, and the output writers. - FFmpeg — Resampler options —
async,min_comp,min_hard_comp,max_soft_comp,first_pts, with defaults. - FFmpeg Doxygen —
AVPacketandAVStream— the canonical PTS/DTS andtime_basedefinitions. - Matroska — technical notes —
TimestampScale, cluster-relative block timestamps, and the one-millisecond default tick. - ITU-R BT.1359-1 — Relative timing of sound and vision for broadcasting — the standards reference for acceptable audio/video offset.
- Leandro Moreira — Digital Video Introduction — the best free grounding in frame types, GOP structure, and why any of this is necessary.
- Wikipedia — Video compression picture types — I/P/B frames and GOP terminology.
Sync problems feel mysterious because the failure is visible and the cause is invisible: two integer sequences, in two different rational units, describing streams that were never mechanically bound to each other in the first place. Once you can read those integers, the mystery collapses into a short list of measurable failure modes — and a much shorter list of correct fixes. If a computer that struggles to keep up during capture is part of your problem, that bottleneck is worth fixing before any flag is.
📚 Recommended Resources
Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.
Start with the free material. For timestamps specifically, the primary sources are better than any book on the shelf, and they cost nothing:
- FFmpeg’s Doxygen reference — the
AVPacket,AVFrame, andAVStreampages are where the PTS/DTS invariant and thetime_baserational are actually defined. Two minutes here settles arguments that blog posts have been having for a decade. - Digital Video Introduction — Leandro Moreira’s free, illustrated walkthrough of frame types, GOP structure, and why the reorder in this article has to exist. The best free grounding in video coding, and the rare one that treats AV1 as first-class.
- Matroska’s technical notes — the
TimestampScaledefault and cluster-relative block timestamps, from the format’s own spec. This is the document behind the millisecond-tick problem above.
Only two books genuinely earn a place next to those.
Video Demystified: A Handbook for the Digital Engineer
Keith Jack · 5th edition, Newnes, 2007 · Paperback, 944 pp
The reference handbook for the plumbing layer this article lives in — video signal formats, the MPEG systems layer, and the transport-stream machinery where the 90 kHz clock and its timestamps come from. It is the closest thing the printed literature has to a book about this problem. Be clear-eyed about the date: a 2007 handbook predates HEVC and AV1 entirely, so read it for the systems-and-timing fundamentals, which have not moved, rather than for codec coverage.
Digital Video and HD: Algorithms and Interfaces
Charles Poynton · 2nd edition, Morgan Kaufmann, 2012 · Hardcover, 752 pp
The book that explains where 30000/1001 came from and why frame rates are rationals rather than round numbers — the historical and engineering backstory to the time-base arithmetic in this article. It is a color and signals reference, not a codec text: roughly one part in six touches compression, and there is no HEVC or AV1 in it. Buy it to understand the signal, and pair it with the free specs for everything else.