Posted on by & filed under go, python.

This post will describe my experience porting a nearest-neighbor web service from Python to Go. Rather than discussing Go’s language features or best practices, which I am not qualified to do, I will try to entertain you with an in-the-trenches look at the porting process. We will enjoy some first-blush encounters with Go programming from a Python programmer’s perspective, such as:

  • Dev environment
  • Third-party libraries
  • Testing
  • Performance profiling
  • Deployment


Though I was already interested in Go because of its concurrency primitives and apparent simplicity, I was inspired to try a Python port after going to OSCON this year, where I attended all of the Go talks (the truth comes out!).


I am pretty new to Go, so you may see anti-patterns and weak use of language features. For example, I have not used Go’s interfaces or concurrency natives in this code.

For some good slides from an experienced Go programmer, see the OSCON 2013 talk “Go Best Practices” by Compoy Flores.

About the Web Service

The web service is a weekend project that I originally wrote to see how easy it would be to perform a nearest-neighbor search through > 50,000 locations without relying on a database (repo links at the bottom).

Basically, given a coordinate in Portland, Oregon, it uses a kd-tree to find crimes committed within a half-mile of the location. For an explanation of kd-trees and nearest-neighbor queries, see “Nearest Neighbor Queries.”

My biggest gripe with the Python version was that it required SciPy, so I had a ten-mile requirements list that included a Fortran compiler and a custom Python buildpack for Heroku.

I was hoping to simplify deployment with a Go version, get a speedup and check out the language.

Go’s Syntax

I won’t get into Go’s syntax too much, but let’s do a simple spot-check. Here is some Go code:

points := finder.All().Points()
nodes := make([]*kdtree.Node, len(points))
for i, p := range points {
        n := kdtree.Node(*p)
        nodes[i] = &n
finder.Tree = kdtree.BuildTree(nodes)

So, “:=“ sets new variable without declaring it, “make” creates slices (dynamic arrays), capital letters in function names refer to public attributes, curly-braces exist, * dereferences a pointer and & creates one.

If you threw up in your mouth, that’s ok. I was a hater too, but after using the language for a while the syntax and features grew on me, just like Clojure grew on me in a former life. Go feels powerful and simple to write.

That’s all the time I’ll spend on the language itself. This is a nice overview if you are interested.

Dev Environment

Installing Go

Installing Go was pretty easy. I used Homebrew on my Mac, but you can find OS-specific instructions on the Go web site.

I will note one thing, however:

  • You only need GOROOT if you install Go somewhere other than /usr/local/go or c:Go (source)

Creating a Workspace (like virtualenv)

After I had a “go” command on my path, the next step was figuring out how to compile something with it. This was a little tricky because I had used Go once a few years ago, and the required environment variables had changed. I will spare you the boring story and summarize:

  • You need a GOPATH environment variable to compile code
  • You can point your GOPATH at a directory that you share for all your Go projects, maybe in ~/go or something — this is akin to using the system site-package (shudder)
  • Or, you can separate dependencies by creating a directory for each of your Go projects and setting GOPATH to that when you work on it — this is like using virtualenv
  • Creating a symlink in GOPATH that points to your source code in a different directory tree did not work for me

You can find more details about the workspace in the Go documentation.

This all felt a little janky. With Python, I use virtualenv and virtualenvwrapper to easily setup new dev environments for projects and switch between them, and the environments can be separate from the code — e.g., my envs are in ~/envs and my source is in ~/src.

With Go, the directory that my project code is in seems long. Inside of the workspace, you need a “src” directory, and inside of that, the packages should be properly name-spaced, so I ended up with ~/src/radar/src/ Not joyful!

After I figured out all this mundane trivia, I began to have more fun.

Editor Plugins

In short: syntax highlighting abounds; deeper mysteries remain hidden.

I was open to using whichever text editor had the best Go plugin. I even looked at Acme because some of the Go devs use it, but it wasn’t my cup of tea.

Long story short, the Vim plugin I found worked pretty well.

The various other plugins I tried (for Sublime Text 2, Intelilj, etc.) may all work better now than they did a few months ago when I set up my environment.

Replacing Third-Party Libraries


I was hopeful to find a Go replacement for SciPy’s kd-tree implementation because SciPy was a pretty heavy requirement for my tiny Flask web service.

After a lot of poking around, I came up with biogo.kdtree. There were a couple of others that seemed like weekend projects, whereas biogo looked pretty solid. The package even had tests!

Skip ahead about twelve hours and I had not managed to write any code that successfully performed a nearest-neighbor search using biogo.kdtree. It still looks like an awesome library, so I chalk this up to the fact that I’m not a scientist.

I settled on a package that I overlooked on my first pass: the “kdtree” package written by Graeme Humphries. Taking another look, it seemed legit. I loaded it up and was querying a tree pretty quickly.

The best part was that this kd-tree library had no third-party dependencies!

Here is what searching the SciPy kd-tree looked like:

distances, indices = self.crime_kdtree.query(point, k=max_points,

That’s pretty clean to my eye. Here is the equivalent in Go:

ranges := map[int]kdtree.Range{
    0: {query.Coordinates[0] - HALF_MILE, query.Coordinates[0] + HALF_MILE},
    1: {query.Coordinates[1] - HALF_MILE, query.Coordinates[1] + HALF_MILE}}
results, err := t.Tree.FindRange(ranges)

Notice that this is a typed language and Go does type inference. It’s not bad. After working with Go code a while, I started to enjoy the types (probably too much).


For web routing, I first looked at Revel. This seemed more like Django than Flask. Revel looked cool and I would choose it for bigger web projects, but I only needed routing.

Next I considered Gorilla Web Toolkit. GWT is even more stripped down than Flask! You can use just the router if you want — mux — which is what I did. So far, that is all I’ve needed.

To use mux, you create a Router and pass it off to http.Handle to intercept all incoming routes. This is what it looks like for a route that takes two float coordinates:

r := mux.NewRouter()
r.HandleFunc("/crimes/near/{lat:[-+]?[0-9]*.?[0-9]+.}/{lng:[-+]?[0-9]*.?[0-9]+.}", handler)
http.Handle("/", r)

In the request handler for this route, you can pull out match groups, e.g. “lat” and “lng” from a map, which is like a Python dictionary.

Also, one of the nice things about mux is that you can name routes, like in Django, and reverse them (see the docs).


Go ships with “testing” package in the standard library, so I used that. Coming from Python’s unittest, I was dismayed about this style of testing. Here’s an example:

func TestCrimeFinderNewCrimeFinder(t *testing.T) {
    finder, err := NewCrimeFinder("data/test.csv")
    if err != nil {
        t.Error("Error creating CrimeFinder: ", err)
    if len(finder.CrimeLocations) != 224 {
        t.Error("Wrong number of CrimeLocations: ", len(finder.CrimeLocations))

There is no suite, so as you can see, I don’t know quite how to name these nasty-looking tests. There are no friendly helpers like “assertEqual,” “assertIn” and the like. So, your tests with the testing package involve lots of if statements. Again, I was a hater at first, but it’s not as terrible as it looks (it’s still kind of terrible).

I have another Go project in mind, so when I start that one I’ll explore an alternative test framework like gocheck.

Performance Profiling

After toiling away in that uncomfortable state a mildly-experienced programmer achieves when writing in a new language, I produced … something. Then I ran it and benchmarked with wrk.

I don’t have the output from those runs, but on a pretty smokin’ machine, it only handled a few hundred requests. The first performance problems were easy to spot: I was building a new kd-tree for every request (WTF), among other atrocities. After nailing those down, though, it was still slow.

I read through a blog post on how to profile Go programs.  Then I sampled while under load from wrk. There were some odd things, like the fact that the json package used a language feature called reflection to dump structs to maps. This was the slowest thing in the code, other than the kd-tree search! I built the strings myself using buffers for a speedup.

The interactive profiling tool go tool pprof was just as handy, perhaps even better, than some of the profiling tools I’ve used on Python programs.


I mentioned in the introduction that I did not use any of Go’s concurrency features in my code. While that is true, I benefitted from library code that did support using multiple cores. Check out what happens to my benchmarks if I gradually increase the number of CPUs Go uses with the GOMAXPROCS environment variable:

  • With 1 CPU, I got 496.63 requests/sec.
  • With 2 CPUs, I got 865.45 requests/sec.
  • With 4 CPUs, I got 1202.54 requests/sec

Clearly, the program is scaling up with the number of CPUs that I give it. Nice!


It was easy enough to go get my repo from a Linux server, build it, and run the radar command.

Deploying to Heroku was also a snap, after I cleaned up the layout of the project to follow Go conventions.

With Heroku, you create an instance using a custom Go buildpack, create a Procfile to start the server, and push-to-deploy as usual. Here is the extent of my Procfile:

web: radar -p $PORT -f data/crime_incident_data_wgs84.csv

A Warning About File Paths

This is just a note for Python people who don’t do much compiling (like me). Originally I wrote the server to join the user-supplied data filename to the current directory that radar.go was in at compile time. This didn’t work on Heroku because the file is apparently compiled in a different location from the current working directory of the shell that runs the Procfile command. Makes sense, in retrospect.


I enjoyed writing Go. Here are some pros:

  • I felt like I had more control over performance than with Python — no need to drop into C!
  • Deploying was super easy
  • My Go binary had a lower memory footprint than the Python version (100 mb versus several hundred)

And some cons:

  • Third party library ecosystem is nowhere near as complete as Python’s (as expected)
  • I did not magically achieve a lightning-fast program
  • It took more than 15 minutes for me to figure out how to deploy to Heroku (this one’s really on me for being lazy)

And here is some terrible code to peruse for laughs:

  • Original Python version (unsafe, poorly tested, abominable):
  • Go version:

Tags: benchmarking, Go, haters, Heroku, laziness, Python, scipy, testing, what is a kd-tree i have no idea,

3 Responses to “Porting a Python Web Service to Go”

  1. astaxie

    If you are writing a json server, I think maybe beego will help you. In beego there’s many helpful modules to enhance your application,such as session, logs, cache, monitor, orm, validator and so on. I hope you have fun with beego.

  2. Adrian

    One adminstrative comment: Your workspace path only needs to be $GOPATH/src/; unless ~/src/radar is your $GOPATH.


  1.  Porting a Geographic Nearest-Neighbor Python Web Service to Go