Twenty Years of Pandoc

On August 3, 2006, I uploaded the first version of pandoc to my website, releasing it under the free GPL license. Pandoc 0.1 consisted of about 3000 lines of Haskell code, with no dependencies aside from GHC’s standard library. It could convert Markdown, reStructuredText, HTML, and LaTeX documents into any of these formats, plus RTF or S5. I had no idea at the time that this would just be the first of over two hundred releases over the next twenty years; that the project would become the most popular program written in Haskell ; that I would spend countless hours on bug-fixes, improvement, and project management; that I would collaborate with programmers in many other countries; that pandoc would come to support over fifty document formats; that it would allow automatic generation of citations and bibliographies; that it would become integrated into academic writing tools like Quarto and Jupyter Notebook ; that it would be installed on millions of computers around the world.

How did this happen? I want to take advantage of pandoc’s birthday to tell the story of the project, as best I can remember it.

People often ask: Why is pandoc written in Haskell? There could have been good answers to this question: Haskell is a very good language for writing this kind of application. But in fact, I didn’t decide to write a document converter, then decide to use Haskell for it. I decided to use Haskell, and then decided to write a document converter in it.

I had heard about Haskell from the blog of a philosophical logician friend, Greg Restall . Of an introductory book on Haskell, he said: “I’m glad that this wasn’t the textbook in my introductory computer science course, long ago in 1986. If it were, I may have fallen in love with computing and never become a philosopher” ( consequently.org ).

Intrigued by this (and not heeding Restall’s warning about the potential effects on my future philosophical productivity), I read A Gentle Introduction to Haskell to get a basic understanding of the language. But the only way to really learn a programming language is to write something in it. I saw that Haskell was good for writing parsers and compilers, and it came with a really nice parser combinator library (parsec), so I decided to write a Markdown parser.

At that time, there were implementations of Markdown in Perl, Python, Ruby, and PHP; they all transformed Markdown directly to HTML through a sequence of regex transformations. Pandoc took a different approach. It parsed the Markdown using parser combinators and produced a real abstract syntax tree (AST) , which it could then render to HTML or another format. This was a more reliable architecture (avoiding many quirks of the regex versions). It was also a more extensible one: by writing N parsers (“readers”) and M renderers (“writers”), one could support N × M conversions. Soon I added a reader for reStructuredText, because I kept a lot of my lecture notes and handouts in that format. And I added a writer for LaTeX, because I wanted to be able to produce PDFs. Then I added a writer for Markdown, so I could start to convert my reStructuredText notes to Markdown. And from there the project just snowballed.

Thus, a project that started out as nothing more than the product of procrastination was nurtured by the joy of writing in Haskell and by its increasing usefulness for my own academic work.

In August 3, 2006, I decided to make the source code available on my website. By now pandoc supported HTML, LaTeX, RST, and Markdown as input and output formats, and RTF as an output format; also PDF via LaTeX.

I made no attempts to advertise the project, other than emailing two friends. This was before social media (which I’ve never used anyway), before GitHub, and before Hackage, the Haskell package repository. But apparently some people stumbled across it on my website and started using it. In October I was contacted by a Turkish developer, Recai Oktaş, who was trying to get certified as a Debian developer and wanted to package pandoc for Debian linux. So I worked with him to do that. This was a great learning experience for me and it greatly increased the visibility of the project.

During 2007, I continued to improve pandoc, largely guided by my own needs. Version 0.3 added the DocBook writer and the now-standard syntax for footnotes in Markdown. Version 0.4 added support for Markdown tables, definition lists, super/subscript, strikeout, and enhanced ordered lists, as well as writers for groff man pages and ConTeXt. This was the first release to go on the Hackage Haskell package repository, which was started in 2007. The Hackage archive and the new cabal-install tool, which automatically resolved and fetched dependencies, opened up the possibility of depending on external packages.

Pandoc 1.0 was released in September 2008, with new writers for MediaWiki, GNU Texinfo (contributed by Peter Wang), OpenDocument (contributed by Andrea Rossato), ODT, and delimited code blocks (now called “fenced”) with automatic syntax highlighting. Support for ODT requires the ability to create a zip archive, and at the time there was no Haskell package for this, so I created one ( zip-archive ), using the excellent binary package for binary parsing and serialization. Support for syntax highlighting required a syntax highlighting library, which also did not exist in Haskell. For this, I wrote highlighting-kate , which parsed the XML syntax definitions used by the Kate text editor and turned them into Haskell code highlighters. This allowed pandoc to support a large number of syntaxes right off the bat. This version also contained support for automatic generation of citations and a bibliography using CSL style , using Andrea Rossato’s citeproc-hs library.

Throughout this period, I was involved in discussions with other Markdown implementers on the (now defunct) markdown-discuss mailing list . The syntax for delimited code blocks, which pandoc supported long before GitHub popularized fenced code blocks, was worked out in collaboration with Michel Fortin, the maintainer of PHP Markdown Extra. I took care when adding extensions to pandoc’s Markdown to pay attention to prior art, for example copying PHP Markdown Extra’s definition list syntax. During this period, I also became aware of many ambiguities in Markdown’s syntax—a situation I would later try to improve in the commonmark project.

The next big change to pandoc came in version 1.4 (released in January 2010), which introduced a flexible template system, replacing hard-coded headers and making pandoc’s output much more customizable.

In 2010, we moved from Google Code to GitHub , which would do even more to increase the visibility of the project. Further releases in 2010 and 2011 added support for EPUB output, Org-mode output (due to Puneeth Chaganti), and Textile input (due to Paul Rivier). Pandoc also gained support for converting TeX math to MathML (for DocBook or HTML), via my texmath library.

Pandoc 1.9, published in 2012, finally made it possible to produce Word docx output. To handle the equations properly, I added support for Word’s OMML format to texmath . This release also added an AsciiDoc writer and support for Beamer and DZSlides, and in 1.9.3 we gained a DocBook reader (with contributions from Mauro Bieg, who became a long-time contributor).

In 2013, we focused on several features that made pandoc much more flexible and customizable. The first was a fine-grained system of Markdown “extensions,” allowing support for the many variants of Markdown that were then proliferating. The second was the ability to include YAML metadata blocks in Markdown, with arbitrary structured fields that populate template variables. The third was the ability to create custom writers in Lua , allowing ad hoc output formats to be supported by users. The fourth was the introduction of JSON filters —user-created programs that transform a JSON serialization of the pandoc AST, allowing the document to be customized between the parsing phase and the rendering phase. Citation processing was moved from the core of pandoc into an external filter, pandoc-citeproc .

This era saw the addition of reveal.js, EPUB v3, DokuWiki, and FictionBook2 output; OPML input and output; and Haddock and MediaWiki input. Notable contributors include David Lazar (Haddock) and Sergey Astanin (FictionBook2).

The year 2014 saw the arrival of three new contributors who would go on to make many contributions to the project. Albert Krewinkel added support for Org-mode input; Jesse Rosenthal added a Word docx reader (complete with track-changes awareness); and Matthew Pickering (at the time a student at Oxford whom I “advised” as a Google Summer of Code Student) added support for EPUB and Txt2Tags as input formats. Supporting EPUB input required being able to convert MathML equations, so Pickering also worked on texmath . We were in very different time zones, and I remember waking up every morning to find all the work Pickering had done during the night. (Pickering has gone on to become one of the core maintainers of the ghc compiler.) All of these contributions were released in pandoc 1.13, together with Clare Macrae’s DokuWiki writer.

Since 2012, I had been involved in a working group that aimed to produce an unambiguous specification of Markdown’s syntax, initiated by Jeff Atwood and including representatives from GitHub, Reddit, and Stack Overflow. The group held intensive discussions in 2012, which petered out in 2013. I still believed in the project and didn’t want to let the work we’d done go to waste, so I sat down in August 2014, before the academic year began, and wrote up a spec for Markdown, as well as parsers in JavaScript and C. I sent the draft spec to John Gruber for comment and did not get a response, so a few weeks later we posted the spec. At this point, Gruber strongly objected and demanded that we not call the project “Standard Markdown,” so we changed the name to “commonmark.” The project has been a success, in that with a few exceptions, most Markdown processors implement the commonmark spec for their core rules. (Commonmark does not concern itself with extensions.)

Pandoc 1.14 (2015) added support for commonmark and a number of extensions (at first via bindings to the C library libcmark , but later, in 2020, via my Haskell packages commonmark , commonmark-extensions , and commonmark-pandoc ). I intend eventually to replace pandoc’s legacy Markdown parser with a commonmark core, but there are still a few key extensions that have not been implemented, so pandoc users must still choose between parsing their documents as markdown (Markdown with pandoc’s extensions) or as gfm or commonmark or commonmark_x (commonmark with a number of extensions). Ironically, although I was the author of the commonmark spec, pandoc still uses a pre-commonmark Markdown parser!

The next year brought some important changes in the pandoc AST, with the addition of image and link attributes, a SoftBreak element (enabling pandoc to preserve line breaks from the original source, or wrap, depending on a command line setting), and a LineBlock element. MarLinn added an ODT reader, Chris Forster added a TEI writer, and Ivo Clarysse added support for DocBook 5.

Pandoc 2.0 (released in 2017) brought some big architectural changes, worked out in collaboration with Jesse Rosenthal. In the past, most of pandoc’s readers (parsers) and writers (renderers) had been “pure” (that is, they had Haskell types that prevented them from having any side effects, including I/O operations). But some formats needed to be able to do I/O for a fully faithful conversion. (For example, reStructuredText has a syntax for including files, so the parser needs to be able to read files; in some other formats, images require explicit sizes, so a renderer has to be able to read image files, perhaps fetching them using HTTP, and determine their sizes.) We designed a system that allowed