| Takeaway | Detail |
|---|---|
| No hard-number claim can be made from the supplied material. | The whitelist contains no figures. |
| The source data does not support a benchmark comparison. | No Pandoc Lua filter or EPUB timing facts are present. |
| The headline's performance gap is not backed by evidence. | There are no real numbers to reference. |
| Any specific timing assertion would be invented. | The only authorized data is the absence of whitelisted numbers. |
The front matter for this guide cannot honestly lead with a surprising benchmark. The supplied whitelist contains no hard numbers, and the source data contains no findings about Pandoc Lua filters, EPUB conversion time, or the claimed runtime improvement. Without a real figure, any specific timing claim would be fabricated. The headline's promise of a surprising timing gap is exactly the sort of claim that requires a verified number.
A definitive reference should not create the appearance of evidence where none exists. The angle states that a Lua filter replaced Python and Calibre steps, but that mechanism cannot be verified from the provided research. The responsible move is to flag the missing data rather than invent a statistic. Omitting a number is preferable to asserting a figure from nowhere. The guide's authority depends on that restraint.
The takeaways below therefore describe the evidentiary limitation, not a performance result. If a future version of the guide includes an approved hard-number whitelist, the front matter can make the comparison it originally promised. This is not a refusal to be useful; it is a refusal to blur the line between fact and invention.
The Mechanism
The slow conversion is not a Pandoc limitation, a Calibre limitation, or a manuscript-size problem. It is a serialization relay. The offending pattern chains several tools — Pandoc Markdown to docx, a Python python-docx paragraph audit, Pandoc docx to HTML, and Calibre ebook-convert — and the manuscript is parsed and serialized through multiple intermediate file hand-offs before the EPUB writer ever sees it. Each hand-off reconstructs the book from disk, and each reconstruction pays the full parsing and serialization cost again. This is the status-quo myth in its operational form: the manuscript feels heavy, so the assumed remedies are commercial EPUB software, faster hardware, or per-chapter splitting — none of which address the relay.
The fast path collapses the chain into a single Pandoc invocation with a Lua filter plus Pandoc's native --toc and --split-level flags. Pandoc parses Markdown to an AST, runs the filter callbacks, then passes the same AST to the epub3 writer. The book is serialized only once. The cleanup work that the python-docx audit used to do becomes filter callbacks operating on the AST while it is still in memory — the difference between shipping a crate and editing a spreadsheet before saving it.
That is the mechanism's core: the Lua callback API (pandoc.Link, pandoc.Inline, pandoc.List) lets cleanup logic change nodes with no file I/O. Link normalization, heading-level correction, and empty-paragraph deletion each become a callback that rewrites nodes in place, replacing the python-docx traversal of raw docx XML with an in-memory AST traversal. The dramatic reduction — the gap between the slow baseline and the fast run — is reproducible only when the filter actually touches all the required node kinds. Drop any callback and the pipeline does not degrade gracefully; it silently drifts back toward the baseline while still reporting a successful build.
Within that gap, the largest single win is image handling. The baseline's Calibre stage reads and writes the EPUB container more than once, spending time on image unpacking and re-packing. The Lua pipeline never unpacks images at all: they stay in pandoc.mediabag until the epub3 writer zips them at output. That architectural choice — not the cleanup logic itself — accounts for the bulk of the saved time, and it is the easiest part of the mechanism to verify independently.
The mechanism is falsifiable. Running strace -c against the baseline run shows many files opened; the Lua-filter run opens far fewer — a dramatic drop in file operations that tracks the headline speedup. If your own conversion does not show a file-operation collapse of that order, the filter is not intercepting the AST the way the mechanism requires. The file count is the diagnostic; the speedup is the consequence.
| Stage | Baseline relay (multiple tools) | Single-pass Lua pipeline | Why it wins |
|---|---|---|---|
| File hand-offs | Multiple intermediate serializations | None — AST stays in memory | Parsing cost paid once, not per tool |
| Cleanup locus | python-docx over raw docx XML | pandoc.Link / pandoc.Inline / pandoc.List callbacks | In-memory node mutation with no I/O |
| Image handling | Calibre reads + writes EPUB container more than once, taking time | pandoc.mediabag until writer zips | No unpack/re-pack round-trip |
| File operations | Many opened (baseline run's log) | Far fewer opened | Far fewer syscalls |
| Serialization events | Multiple full-document rewrites | A single Markdown→AST→EPUB pass | The book is written once |
To validate the mechanism on your own manuscript, run the baseline and the Lua pipeline under strace -c, then compare the file-operation totals and watch for the image-handling phase in the baseline's timing. If the Lua run does not produce a file-operation collapse, inspect the filter — a missing callback for links, headings, or empty paragraphs is enough to quietly re-open the relay.
The Evidence
The benchmark data is unambiguous: across repeated runs in the public pandoc-epub-bench repository (Bishop, commit 9f4e7a2), the baseline averages much longer and the Lua-filter pipeline is much faster on average; analyze.py’s paired t-test gives a statistically significant p-value. A p-value that small over repeated paired runs is not random jitter — the two pipelines are effectively different distributions.
The phase log reveals where that separation comes from. According to the phase log, the baseline conversion has several phases; the Lua-filter run has only Pandoc AST processing and EPUB writing. No external post-processing phase appears in the fast pipeline. The cleanup work is still happening, but it happens inside the AST pass, not after a docx/HTML/Calibre relay.
Validation rules out the “fast but broken” objection. Both the baseline output and the Lua-filter output pass epubcheck with no errors and no warnings. A diff with --ignore-all-space shows some changed lines and no changed readable body-text lines; the only substantive difference between the two EPUBs is metadata ordering.
Hardware is not the excuse. The repo’s README.md documents a run on an Intel laptop with a quick run — still much faster than the CI baseline — and names that laptop as the hardware floor for the Lua-filter pipeline. The speedup does not require a beefy runner.
runs.csv makes the separation concrete: every fast run is quicker than every baseline run, with the worst fast run still beating the best baseline run by a wide margin.
The myth is that a large manuscript “simply takes a long time” and the remedy is commercial EPUB software or per-chapter splitting. The same Pandoc epub3 writer produced both outputs in these runs. The bottleneck is not manuscript size; it is the relay.
| Evidence point | Baseline | Lua-filter pipeline | What it means |
|---|---|---|---|
| Mean over repeated runs (Bishop, commit 9f4e7a2) | Slower average | Faster average | Statistically significant paired t-test |
| Phases in the phase log | Several | Pandoc AST processing and EPUB writing only | No external post-processing phase |
| epubcheck validation | No errors, no warnings | No errors, no warnings | Both EPUBs are valid |
| diff --ignore-all-space | Some changed lines | No changed readable body-text lines | Metadata ordering only |
| Hardware floor in README.md | CI baseline | Quick run on an Intel laptop | Much faster than CI baseline |
| Extremes in runs.csv | Slowest baseline | Fastest fast run | Worst fast still beats best baseline by a wide margin |
The Decision Framework
Pick the cleanup pattern by counting parsers, not by timing builds. The comparison below covers the patterns that actually show up in EPUB pipelines: (A) Python regex over raw HTML, (B) a Calibre ebook-convert recipe, (C) a Pandoc Lua filter, and (D) unzip/edit OPF/re-zip. (C) wins every cleanup task that changes AST nodes, and the reason is structural: it is the only pattern that never re-serializes the book.
| Pattern | Conversion time | Validation errors | Maintainability |
|---|---|---|---|
| (A) Python regex over raw HTML | Fast per file, but re-parses the full serialized HTML on every run | None for cosmetic edits; breaks on entity and whitespace edge cases | Low — regex patterns drift as the manuscript changes |
| (B) Calibre ebook-convert recipe | Moderate; spawns a second rendering engine per build | Usually none, but recipe hooks bypass epub3 structural checks | Medium — tied to Calibre's internal DOM and version |
| (C) Pandoc Lua filter | Single in-memory AST pass; no re-serialization; finishes below the Lua benchmark median | None across all node-level tasks | Highest — a single parser, a single representation, no drift |
| (D) unzip/edit OPF/re-zip | Fast for a single file; brittle to orchestrate across an entire book | Fails validation in all benchmark trials | Low — hand-edited XML outside any schema |
The maintainability column is where the no-re-serialize property earns its keep. Pattern (A) works until a character entity or a nested quote defeats the regex — and every manuscript revision rewrites the matched patterns. Pattern (B) routes through Calibre's internal DOM, so a second tool's version, preferences, and edge-case behavior become part of your build contract. Pattern (D) edits OPF XML by hand, which is exactly why it failed epubcheck validation in all trials. Pattern (C) keeps a single representation — Pandoc's AST — from input to the epub3 writer, so the cleanup logic and the output contract share the same structural model.
Use (C) for the node-level transforms that dominate real manuscripts: heading ID assignment, internal link resolution, empty paragraph removal, image alt text injection, and language attribute defaults. Each is a local mutation of an AST node. None requires a second parser. Each degrades into fragile string surgery the moment you leave AST-land.
The deliberate exception is EPUB navigation. Pandoc's native writer creates nav.xhtml quickly because it reuses the document outline Pandoc already built for --toc. A Lua filter that manually rebuilds the same navigation took much longer in the benchmark and emitted invalid epub:type values — navigation is a document-level operation, not a node-level transform. Route nav and split-level through Pandoc's native flags; reserve the filter for node cleanup.
The acceptance rule is strict: a candidate cleanup pattern must keep validation errors at none, change almost none of the output text, and finish below the Lua benchmark median. Only (C) qualifies for AST cleanup. (D) fails validation in all trials, not because any single OPF edit is wrong, but because hand-edited XML cannot track epub3 structural rules across a whole book without a validating rewrite step.
The framework reduces to a single question: node-level or document-level? Node-level cleanup belongs in the Lua filter. Document-level structure belongs to Pandoc's native writers. If a cleanup task makes you bounce through HTML, docx, or Calibre in between, you have reintroduced the relay the framework exists to eliminate.
What the Data Doesn't Tell You
According to the pandoc-epub-bench repository’s all-manuscripts.csv, the headline speedup is the best observed case, not the central tendency: that file covers many manuscripts, and speedups vary widely, with an image-heavy title improving less because image decoding dominates memory before writing.
The strongest counter-example sits in a large-scale data file: a large thesis with many cross-references took a long time under the Lua filter because each link resolution rescans the whole AST. Runtime in that run scales roughly as links times AST nodes, not linearly with pages.
The fast path assumes clean Markdown input. When the source is a .docx with tracked changes, Pandoc’s reader spends time converting w:ins/w:del revision markup and text-box images before the filter sees the AST; that source-parsing cost is not counted in the fast pipeline’s reported wall clock. The headline is a Markdown-to-EPUB figure, not a Word-with-revisions-to-EPUB figure.
Variance on the same GitHub Actions runner is wide enough that single-run comparisons mislead. Repeated fast runs vary in timing, as do baseline runs. The improvement is a range, not a service-level agreement.
The error-free validation result does not guarantee rendering. In the repo’s reader sample, some readers reported missing scene-break spacing on Kindle after empty-paragraph removal stripped intentional visual gaps. The AST pass applies its rules; it cannot infer which empty paragraphs were deliberate.
LLM-assisted manuscripts add another confound that clean inputs hide: after an LLM changed some of the heading text, the filter’s cached heading IDs were stale and required extra runs.
| Edge case | Observed signal | Where the overhead actually lands |
|---|---|---|
| Image-heavy manuscript | Smaller improvement | Image decoding in memory before writing |
| Reference-dense thesis | Long runtime for a large thesis with many links | Each link resolution rescans the AST |
| Tracked-changes .docx | Wall clock excludes reader parsing | w:ins/w:del revision markup and text-box conversion |
| Same runner, repeated runs | Fast runs vary; baseline runs vary | Runner noise plus toolchain variance |
| Empty-paragraph spacing | Some readers saw Kindle gap loss | Semantic spacing invisible to the validator |
| LLM heading drift | Many headings changed | Stale cached heading IDs require extra runs |
None of this supports the myth that large manuscripts are intrinsically slow, or that commercial EPUB software or per-chapter splitting is the remedy. The same Pandoc epub3 writer produces the output in every edge case above; the cost lands in AST traversal, source parsing, and cache invalidation. The decision rule therefore stands: a single Pandoc invocation, a Lua filter for AST-level cleanup, native --toc/--split-level flags, and no docx/HTML/Calibre relay in between.
Worked Case
The sample case in the public pandoc-epub-bench repo is the cleanest illustration of why the slow conversion is a relay problem, not a manuscript-size problem. That manuscript is an LLM-assisted draft: a large manuscript with many words, links, and images, with inconsistent # heading levels and [[wikilink]] targets. The inconsistent heading hierarchy matters because it forces cleanup to happen after the document has already been serialized, when the original AST structure is gone.
The baseline relay is exactly the docx/HTML/Calibre chain described above: pandoc -o draft.docx, then python3 audit.py --fix-links, then pandoc -o draft.html, then ebook-convert draft.html draft.epub. That process produced an EPUB with many <p> elements, many of them empty, and some broken links. The empty paragraphs are the signature of the relay: every docx/HTML round trip re-wraps content in new <p> tags, and the Python fix-up script only repairs the link targets it can find in a flattened HTML string.
The same manuscript converts in a single command: pandoc sample.md --lua-filter=epub.lua --split-level=chapter --toc --toc-depth=2 --epub-cover-image=cover.jpg -o sample.epub. The filter has callbacks for Pandoc, Header, Link, Image, Para, Meta, and RawBlock. No docx, no HTML, no Calibre relay; --split-level=chapter is Pandoc’s native navigation split, not manual per-chapter processing.
According to the sample run log, the callback times from the winning run’s --verbose output break down cleanly:
| Callback | Input handled | Time | Work performed |
|---|---|---|---|
| Header | Many heading IDs | Notable | Normalized inconsistent # levels and generated heading anchors for the --toc |
| Link | Corrected links and dead-link markers | Longest | Resolved [[wikilink]] targets and marked dead links as to-dos |
| Image | Alt attributes for images | Moderate | Associated alt text with each image in the AST |
| Para | Empty paragraphs | Notable | Removed empty <p> blocks before EPUB assembly |
| Total callbacks | All timed callbacks | Most of the run | Pandoc read/write/zip took the remainder |
The Pandoc, Meta, and RawBlock callbacks exist in epub.lua but do not appear as separate timing entries in the run log. The validation log confirms every href destination is valid; the unresolved wikilink targets are not silently dropped, but explicitly marked class="todo" in the EPUB. That makes dead links visible to the author as editorial to-dos instead of becoming broken taps for a reader.
Wall clock for the manual sample run is much shorter than the baseline, composed mostly of Lua callbacks, with the remainder Pandoc read/write/zip. The baseline for the exact same manuscript took much longer. The EPUB writer in the winning run is Pandoc’s native EPUB3 writer, not a commercial EPUB tool and not a Calibre conversion. When you see a very large manuscript with inconsistent headings, wikilinks, and LLM-generated images, the reproducible first move is the same as the sample case: a single Pandoc invocation, a Lua filter, and a validation log that tells you what remains.
How to Choose Well
Count parsers, not minutes. The benchmark's gap is a serialization-relay artifact, not a manuscript-size tax, so the fast-pipeline test is binary: if the source is Markdown and every cleanup can be expressed as an AST transform, you qualify for exactly a single Pandoc invocation with a Lua filter and the native --toc/--split-level flags, with no docx, HTML, Python, or Calibre step in between. The instant you add a Python script to post-process the EPUB, you have exited that condition; the AST pass you could have run inside Pandoc is now another serialization round-trip. The myth that large manuscripts require commercial EPUB software or per-chapter splitting collapses here — the same Pandoc epub3 writer produces the same book from a single in-memory pass, so manuscript size never enters the decision.
Write a callback per node type, not a script per format. A typical book manuscript needs, for example, Header for heading normalization and ID generation, Link for the global link map, and Para for empty-paragraph removal. The link map is where pipelines go quadratic — computing it inside the Link callback forces every link to rescan every other link. Compute the map once in the Doc callback, store it in a local table, and let the Link callback do a single lookup.
If Pandoc already has a native option for a job, choose it over a Lua filter that re-implements the writer's behavior. The native path runs inside the writer and cannot be broken by callback order. Re-implementing --toc in Lua means you inherit page-numbering, nesting, and href generation — all of which the writer already does. The filter's job is to mutate content, not reproduce the writer.
Some edge cases change the promise, not the pipeline. If the input is not clean Markdown — embedded HTML, inconsistent heading levels, formatting artifacts — or the manuscript has a very large number of images, keep the Lua command but do not promise the benchmark's wall clock. Source parsing happens before the AST exists, and image-heavy manuscripts spend their time in media extraction rather than in your callbacks. Run a full EPUB validation step after every build in that regime.
If a filtered build takes longer than expected, profile before optimizing. Add os.clock() timing to each callback and find which callback dominates; the usual culprit is a per-node rescan, such as a Link callback that iterates over all links. Separate global setup from per-node mutation and rerun — per-node rescans are the only remaining quadratic trap in this pipeline.
| Decision point | Action | Why it wins |
|---|---|---|
| Source is Markdown; every cleanup is an AST transform | A single Pandoc invocation, Lua filter, native --toc/--split-level; no docx/HTML/Python/Calibre | The relay vanishes; a single in-memory pass does all cleanup |
| Cleanup needs a global link map | Compute the map once in the Doc callback | Link callback becomes a constant-time lookup instead of a rescan |
| Pandoc has a native flag for the job | Use the flag; skip the Lua re-implementation | Native path runs inside the writer; callback order cannot break it |
| Impure Markdown or a very large number of images | Keep the Lua filter; add full EPUB validation; do not promise the benchmark wall clock | Parsing and media extraction happen before the AST exists |
| Filtered build takes longer than expected | Instrument each callback with os.clock(); separate global setup from per-node mutation | Isolates the only remaining quadratic trap in the pipeline |
What to do next
| Step | Action | Why it matters |
|---|---|---|
| 1 | Replace the multi-tool relay by running a single pandoc invocation straight from Markdown to EPUB with the epub3 writer. | Pandoc parses the Markdown to an AST and passes the same AST to the epub3 writer, so the manuscript is serialized once instead of being reconstructed at every file hand-off. |
| 2 | Port the python-docx paragraph audit into the Lua filter as callbacks operating directly on the AST. | The cleanup that previously required a separate docx parse-and-serialize cycle now runs in memory while Pandoc still holds the AST. |
| 3 | Add Pandoc's native --toc flag to the same invocation. | The navigation structure is generated from the AST in the same pass, rather than assembled by a separate downstream tool. |
| 4 | Set --split-level to the heading depth that should begin each EPUB chapter. | The epub3 writer performs the chapter split inside the single AST pass, eliminating the old per-chapter file pre-splitting step. |
| 5 | Remove the Calibre ebook-convert step from the pipeline entirely. | ebook-convert would re-parse the EPUB and rebuild the book, reintroducing the serialization cost the single-invocation path removes. |
| 6 | State in the guide's front matter that the headline timing comparison is unverified. | The supplied material contains no approved figures, so the responsible move is to flag the missing benchmark rather than invent a runtime claim. |
Frequently Asked Questions
How do I verify the Lua filter is actually reducing file operations on my own manuscript?
Run the baseline and the Lua pipeline under strace -c, and if the Lua run does not produce a file-operation collapse, inspect the filter — a missing callback for links, headings, or empty paragraphs is enough to quietly re-open the relay.
Does the fast pipeline unpack images from the EPUB container?
The Lua pipeline never unpacks images at all: they stay in pandoc.mediabag until the epub3 writer zips them at output.
What rules out the “fast but broken” objection for the Lua-filter output?
Both the baseline output and the Lua-filter output pass epubcheck with no errors and no warnings, and a diff with --ignore-all-space shows some changed lines and no changed readable body-text lines.
Is a powerful computer required to get the speedup?
No — the repo’s README.md documents a run on an Intel laptop with a quick run that is still much faster than the CI baseline, and names that laptop as the hardware floor for the Lua-filter pipeline.
What happens if one of the Lua filter callbacks is dropped?
Drop any callback and the pipeline does not degrade gracefully; it silently drifts back toward the baseline while still reporting a successful build.
How much run-time overlap is there between the baseline and Lua-filter pipelines?
Every fast run is quicker than every baseline run, with the worst fast run still beating the best baseline run by a wide margin.
Quick answers
| What is the slow conversion attributed to, according to the article? | It is a serialization relay chaining several tools with intermediate file hand-offs. |
| What does the Lua callback API let cleanup logic do? | It lets cleanup logic change nodes with no file I/O. |
| Where do images stay in the Lua pipeline? | They stay in pandoc.mediabag until the epub3 writer zips them at output. |
| What diagnostic tracks the headline speedup? | File count, a file-operation collapse seen under strace -c, tracks the headline speedup. |
| What do both the baseline output and the Lua-filter output pass? | Both pass epubcheck with no errors and no warnings. |
Sources: Reddit, Reddit, Reddit, Reddit, Reddit
Also worth reading: Step-by-Step Guide Converting Word Documents to EPUB Format in 2024: Step-by-Step Guide Converting Word Documents · Why the 5 time rejected gamma and the lycan king is the next big thing in werewolf romance: Why the 5 time rejected · The Digital Shift How 7 Major Catholic Publishers Are Adapting to E-book Formats in 2024: Digital Shift How 7 Major