mongoDB & Golang == gomongo

Golang’s type based design makes it a language that’s good for BigTable / Document Based Databases.  One of the best Document based Database’s is mongoDB used by some very high profile services.   In this article we look at a great mongoDB driver, gomongo.

Thanks to some wonderful work from mikejs (Michael Stephens) & kless (John Mac) gomongo  is a wonderful implementation of the mongoDB driver that’s so easy to use you’ll be writing documents to the Database in minuets.

A bit about mongoDB

In case you’ve never hear, or only heard very little, about mongoDB, let’s look at what’s so great about it.

Document Based

What’s a Document Based Database? Unlike conventional databases that contain Table’s which have a predefined schema of columns that contain certain “native” datatypes such as string, int, datetime etc.

A mongoDB database has a collection of documents, these documents loosly equate to the Tables found in conventional databases.  However rather than containing a predefined schema of columns the documents can contain any schema.  Additionally, the documents don’t all have to conform to identical schema’s.

The Documents aren’t a “flat” set of columns, that’s to say a “column” on a monogoDB document could contain another column which in-turn could contain another and so on.

The monogDB documents are expressed in BSON -

BSON [bee · sahn], short for Binary JSON, is a binary encoded serialization of JSON-like documents.

Consider the following JSON :

 // First Document
{   _id:1,
    title:"Page Title",
    body:"Some example text"
    comments:[{
        author:"foobar",
        message:"Thanks for the post"
    },{
        author:"hokapoka",
        web:"go.hokapoka.com",
        message:"No problem"
    }]
}
// Another Document
{   _id:2,
    title:"Follow up to Page Title",
    body:"Some other Text"
}

The JSON above defines 2 documents, note how they don’t have identical structures also note the comments property of the first document, it’s an array of objects.

If one of the documents were assigned to myDocument it can be inserted into the database using :

db.foobar.save(myDocument);

Querying mongoDB

If the 2 documents above had been inserted into mongoDB they can easily be retrived via find using any of the values assigned to any property on the document.  For example :

db.foobar.find({ title:"Page Title"});

This would return the first document, of the two, because it matched on the title.

The advanced querying capabilities of mongoDB such as the ability to query deep within the Documents, sort, curosrs and many more features make it a very power database.   Please have a look at the monogDB documentation for greater detail on mongoDB’s features.

Indexing MongoDB

Of course, no database query’s are any good when you have large quantities of data without indexes.  As you would expect mongoDB has a really simple command to enable Indexing on selected properties of documents.

The other features of monogDB are beyond the scope of this article, such as sharding map/reduce and GridFS, there are plenty of great articles available via mongoDB website, I highly recommend reading them.

Who uses mongoDB?

As mentioned above, mongoDB is used by some high profile services that have a huge throughput of data.

Such as bit.ly, SourceForge, foursquare, github and many more.  If mongoDB can be scaled to supporting services like these, it’s got to be given some respect.

Building gomongo

Assuming you have a mongoDB setup, if you don’t it only takes a few minuets – checkout the mongoDB quickstart.  Download the source for gomongo from github so you can make & install the packages.

Before you can build the gomongo package you’ll need download & build the rand package from kless’s  freecrypto github source.

Update : kless has been kind enough to add some background to freecrypto, a better installation method for both gomongo and optionally howto install the rand package of freecrypto via goinstall.

“the goal is to have a cryptography library without copyright to avoid the virality about regulations from USA” – kless

The freecrypto library is a set of independent modules that can be installed and used in programs, such as the gomongo driver.

To install the gomongo driver you can just use goinstall like this:

$ (sudo -E) goinstall github.com/mikejs/gomongo/mongo

*Please note only use the sudo command if it’s to only be installed under root.

Please read kless’s comment below for details on how install just the rand package of freecrypto if you want to use it in your own project.

Build & Install gomongo +freecrypto (Original post, use kless’s guide above)

Once you have the freecrypto source downloaded cd to the rand directory within the source and if you have a configured go environment you’ll be able to build & install just like any other Go package.

make && make install

This should build without any error’s, once you have done that you can cd to the mongo directory of the gomongo source and do the same again.

Once you have done this you’re ready to start accessing your mongoDB from golang.

Using gomongo

You may have noticed the package name when gomongo was built, however if not open  the Makefile located in the mongo directory of te source you’ll see the TARG value is set to : "github.com/mikejs/gomongo/mongo" this is what you will need to import in order to use the gomongo package in your go program.

Update: http://go.hokapoka.com/golang/gomongo-unmarshal-issues-with-types-that-contain-slices/

Add a simple map[string]string document to mongoDB

Let’s start by inserting a map[string]string into a collection on your mongoDB server.

package main

import (
	"github.com/mikejs/gomongo/mongo"
	"fmt"
)

func main() {
    conn, _ := mongo.Connect("127.0.0.1")
    coll := conn.GetDB("example").GetCollection("map_string")

    firstDoc := map[string]string{
        "_id":"1",
        "title":"Page Title",
        "body":"Some example text",
    }

    bsonDocIn, _ := mongo.Marshal(firstDoc)
    coll.Insert(bsonDocIn)

    fmt.Println("Inserted Document")
    coll.Drop()
}

The code above attaches to the local mongoDB server, selects the “map_string” Document Collection of the “example” database.

Then using the type map[string]string an instance, firstDoc, is created that contains some of the content declared in the JSON examples above.

Using the mongo.Marshal method the firstDoc map is converted to a var bsonDocIn, type BSON.  The bsonDocIn is then inserted into the “map_string” document collection of the mongoDB via coll.Insert.

And that’s it, the document has been added to the mongo Database, here’s the output from the mongo shell client :

$ mongo
MongoDB shell version: 1.4.4
url: test
connecting to: test
type "help" for help
> use example
switched to db example
> db.map_string.find()
{ "title" : "Page Title", "body" : "Some example text", "_id" : "1" }
>

Note: Neither the database nor the document collection need to exist prior to inserting a document; if either don’t exist it will be created at the point when the first document is added to the collection / database.

Inserting typed values

While using maps are great to an extent, struct’s are much better when creating production applications.  So let’s look at how in insert a struct into a mongoDB using gomongo.

package main

import (
    "github.com/mikejs/gomongo/mongo"
    "fmt"
)

type ExampleDoc struct {
    Title  string
    Body string
    Comments []Comment
}

type Comment struct{
    Author string
    Web string
    Message string
}

func main() {
    conn, _ := mongo.Connect("127.0.0.1")
    coll := conn.GetDB("example").GetCollection("typed_structs")

    firstDoc := &ExampleDoc{
        Title : "Page Title",
        Body : "Some example text",
        Comments : []Comment{
            Comment{
                Author :"foobar",
                Message:"Thanks for the post",
            }, Comment{
                Author: "hokapoka",
                Web : "go.hokapoka.com",
                Message : "No Problem",
            },
        },
    }

    bsonDocIn, _ := mongo.Marshal(firstDoc)
    coll.Insert(bsonDocIn)

    fmt.Println("Inserted Document")
    coll.Drop()
}

The program above is very similar to the previous example.

In order to use a typed struct rather than a map the struct to be inserted are declared and an instance of the struct’s are created.  This instance of the struct is passed to the collections Insert method and that’s it!

Please note, that struct isn’t a “flat” type, it contains []Comment and the instance that has been created via Composite Literal notation contains 2 Comments, just as the JSON example at the top of the page.

You can use the mongo shell client again to test that the document was added.  In the example below the .forEach(printjson) command was used to display the Document in a more human readable fashion.

$ mongo
MongoDB shell version: 1.4.4
url: test
connecting to: test
type "help" for help
> use example
switched to db example
> db.typed_structs.find().forEach(printjson)
{
        "_id" : ObjectId("4c4e89fa7d72893c5c503b6d"),
        "title" : "Page Title",
        "body" : "Some example text",
        "comments" : [
                {
                        "author" : "foobar",
                        "web" : "",
                        "message" : "Thanks for the post"
                },
                {
                        "author" : "hokapoka",
                        "web" : "go.hokapoka.com",
                        "message" : "No Problem"
              }
        ]
}
>

Using Find in Golang

Now we’ve added a typed Document we can retrieve it from the mongoDB, just as easily as it was added.

package main

import (
    "github.com/mikejs/gomongo/mongo"
    "fmt"
)

type ExampleDoc struct {
    Title  string
    Body string
    Comments []Comment
}

type Comment struct{
    Author string
    Web string
    Message string
}

type searchDoc struct{
    title string
}

func main() {
    conn, _ := mongo.Connect("127.0.0.1")
    coll := conn.GetDB("example").GetCollection("typed_structs")

    qFindDoc, _ := mongo.Marshal(&searchDoc{title:"Page Title"})
    bsonDocOut, _ := coll.FindOne(qFindDoc)

    var foundDoc ExampleDoc
    mongo.Unmarshal( bsonDocOut.Bytes(),  &foundDoc )

    fmt.Println(foundDoc.Title + " :: " + foundDoc.Body )
    coll.Drop()
}

This program is very similar to the insert, an additional struct has been created, searchDoc, and an instance of this is created with the value of title to be matched against assigned, title:"Page Title".

Once gomongo has foudn a doc, via “FindOne”, that match the searched value, the BSON value of the match need to be cast into the type that it represents.  The mongo.Unmarshal method is used to achieve this.

Declare a var of the document type found in the mongoDB, and pass the unique pointer to this to mongo.Unmarshal.  Along-with .Bytes(), the []byte of the BSON value.  The mongo.Unmarshal method will instantiate & assign the values from the mongoDB into the var that has been declared.

That’s it, you now have an instance of the Document that you inserted into the mongoDB will all of it’s values assigned.

But why create a type for searching?

If you don’t want to create a struct for each search you can just as easily use a map, for example, the following code will return the same results as the searchDoc type has done :

qFindDoc, _ := mongo.Marshal(&map[string]string{
    "title":"Page Title",
})

Important points to note regarding Exported properties

There are a couple of important aspects to the struct’s and the value’s that are stored in mongoDB that need’s to be highlighted.

In order to expose the properties to the gomongo driver, the properties need to be Exported, hence the struct properties all begin with a Uppercase char.

While these values are Uppercase in Golang, mongoDB uses all Lowercase chars for the property names of documents.  Hence why the output from mongoDB has Lowercase names, instead of the same case that is used for the structs.

As it happens, the structs only need to have exported values in order to Unmarshal the BSON value.  If you were only writing to the monogo DB you could use lowercase for your structs and all would work fine.

This doesn’t in anyway effect the developer, as the gomongo driver handles the conversions.

This could be a bug in the current version of the gomongo driver, let’s hope so.  Other than this little oddity gomongo is a really great driver.

Thanks to all

A special big thanks to mikejs & kless for the great work on gomongo & freecrypto.

Thank you for reading, hope that you found the article easy to follow, if you have any queries or questions please feel free to leave a comment below or email me at hoka (at) hokapoka.com

Go!