Extending RedCloth with remote pygments for syntax highlighting

If you ever used pygments for syntax highlighting, you know it’s a marvelous thing. What happens when you can’t have Python installed on you server (like e.g. Heroku ) and yet still want to use it?

I tried serving pygments’ highlight method using simple HTTP requests on a remote Google Apps instance for use with Textile and the RedCloth gem. Here’s what I ended up with:

Serving pygments via HTTP requests.

In order to have a remote app serving Python exclusive functionality, I needed something cheap, reliable and persistent. Didn’t take me a long time before I chose the Google App Engine. The Google Apps Python SDK allows for simple HTTP requests processing using just a few lines of code. This, combined with using the pygments lexers and HtmlFormatter, lets easily return highlighted code inside the HTTP response:

from pygments.lexers import get_lexer_by_name
from pygments.formatters import HtmlFormatter
from pygments import highlight

def pygmentify(body, language):
  lexer = get_lexer_by_name(language)
  formatter = HtmlFormatter()
  return highlight(body, lexer, formatter)

class MainHandler(webapp.RequestHandler):
    def post(self):
        self.response.headers.add_header('Content-Type', 'text/plain')

The full code for my pygmentify app is available on github.

Mind that it strips the output of the highlight function of all the unnecessary (for me at least) div and pre tags, so you can arrange them however you want on your target site. You now need to set up your Google Engine App, which is as easy as following the steps on their website.

Extending RedCloth itself

Now it’s time for extending RedCloth so it will actually make use of our freshly deployed pygments app. Looking into the code brought the RedCloth::Formatters::HTML module to my attention. All I needed to do was to override some of it’s methods related to the bc. (code block) formatter in Textile:

module RedCloth::Formatters
  module HTML

    # Add html for encloding the highlighted code block
    def bc_open(opts)
      "<div class=\"highlight\"><pre>"

    # Close it
    def bc_close(opts)

    # Process the code by sending it 
    def code(opts)
      lang = opts[:class].nil? ? 'console' : opts[:class]
      request = HTTParty.post(
        :body => {:language => lang, :body => opts[:text]}
      result = request.parsed_response

This allows the conversion of a following code block:

bc(ruby).. def awesome?

As you might have noticed, I grabbed the textile class argument in the brackets for providing the language for the pygments lexer, so if you need it, you can always provide the language somewhere in the block, e.g. separated by a pipe (vertical bar) and parse the _opts[:text]_ accordingly:

bc(my_class).. ruby |
def moar_awesome?

Integrating with a Rails app

So, to start off we need to provide the new Formatters with the @jnunemaker's HTTParty dependency in your Gemfile (or you could just use Ruby's basic NET::HTTP.post_form).

gem 'httparty'

Don't forget to load the pygments/pygments.rb file in your application.rb:

config.autoload_paths += Dir["#{config.root}/lib/pygments"]

Now in your view, you can use RedCloth as follows in your .haml file:

%p= RedCloth.new(`post.body).to_html.html_safe

However posting a form to the external app with each view render seems extremely bandwidth - inefficient. How about we just store to outputted html inside the model?

require 'pygments'

field :body_formatted, :type => String

before_save :format_html

def format_html
 self.body_formatted = RedCloth.new(self.body).to_html

And then use this in your haml view:

%p= @post.body_formatted.html_safe

Should be it.

I hope you’ll find this useful - I did right here on this site. I'd definitely love to hear on how you would improve this code in the comments.