Emil's Neo Thoughts

Graph Databases, Licensing and MySQL

Today marks a new major milestone for Neo4j: we’re making the core graph database - Neo4j Community - available under the same proven open source license as MySQL, the GNU General Public License (GPL).

That means that in every scenario where you can use MySQL for free, you can now also use Neo4j Community for free.

Why Is This a Big Deal?

This is a big deal because the world has lacked a great graph database with no barriers to adoption. The last year has seen tremendous interest in graph databases. We've seen a rapidly growing Neo4j community, an explosion in customer deployments, and many new graph database implementations in the last 12 months.

But we've learnt that the opportunity is still bigger. Some developers found the Neo4j license too restrictive, preventing their teams from using Neo4j. We listened, and that’s what we are changing today.

The graph database opportunity is at least as big as the MySQL opportunity. The best way to make graph databases achieve their full potential is to make a robust and mature graph database widely available under the same license as MySQL.

Wow! Why Are You Doing This?

It’s been almost two years since the first NOSQL conference in San Francisco in June of 2009. These two years have seen an explosion of NOSQL databases and a surge in interest, evaluation, experimentation and usage by everyone from scrappy startups to global enterprises.

Since the graph data model is the richest of all the NOSQL data models, graph databases have the opportunity to be the most useful for most people in most situations, and ultimately grab the largest slice of the NOSQL market.

Most users today store the vast majority of their transactional information in a relational database, with only a few using a NOSQL database:

Database use in most deployments 2011

But when we talk to some of the largest web properties in the world, the situation is more like this:

Database use in high-end deployments 2011

While the Amazon's and the Yahoo!’s of the world remain significant SQL users, NOSQL is also central to their business. Why? Certainly not because they enjoy writing their own databases! They were forced to deal with information of such size and complexity that they simply could not efficiently manage it all in a relational database. Hence they had to build their own custom in-house NOSQL datastores to satisfy business requirements.

Information is growing today at staggering rates in both size and complexity. Last year more new digital information was generated than during all previous years... combined! This information is highly connected, messy and complex. Nonetheless, it contains vast amounts of business value.

Businesses will want to look at this information, analyze it, and make decisions on it – more and more frequently in real time. Over time, this explosion of complex information will force all deployments to deal with increasingly larger volumes and increasingly higher information complexity. This will eventually impact all database-backed deployment out there and they will grow more and more similar to today's high-end deployments: using both SQL and NOSQL equally.

Database use in most deployments over time

Conclusion

Since the graph data model is the richest of all the NOSQL data models, graph databases have the opportunity to be the most useful for most people in most situations, and ultimately grab the largest slice of the NOSQL market.

In other words, graph databases have the opportunity to be ubiquitous. And because of the MySQL-like scale of this opportunity, we feel that the world deserves a robust and mature graph database licensed the same way MySQL is - under the GPL.

Therefore I'm happy to announce the immediate availability of Neo4j Community under the GPLv3. For more information about the new 1.3 release (including huge store support and a new web UI) please see the announcement. For more details about the new licensing, see here.

And finally, feel free to download and check out the leading graph database in the world, Neo4j Community, now available under the GPL.

NOSQL: scaling to size and scaling to complexity

About a week ago, following nosql east in Atlanta, Jonathan Ellis from the Cassandra project published a fantastic overview of the current NOSQL ecosystem. He analyzes 10 popular NOSQL databases along three axes: horizontal scalability, data model and internal persistence design. It's a great read.

The third axis (internal persistence design) may not be terribly relevant for users of NOSQL systems [1] but the position on the first two axes reveal some important underlying assumptions. In particular, it reveals a focus: is this NOSQL project oriented around scaling to size or scaling to complexity? [2]

The four main NOSQL data models

Now, there are four main categories of NOSQL databases today. Before we get into how they differ in focus, let me just quickly run through them and outline a few key characteristics:

Key-Value Stores

• Lineage: Amazon's Dynamo paper and Distributed HashTables.
• Data model: A global collection of key-value pairs.
• Example: Voldemort, Dynomite, Tokyo Cabinet

BigTable Clones (aka "ColumnFamily")

• Lineage: Google's BigTable paper.
• Data model: Column family, i.e. a tabular model where each row at least in theory can have an individual configuration of columns.
• Example: HBase, Hypertable, Cassandra [3]

Document Databases

• Lineage: Inspired by Lotus Notes.
• Data model: Collections of documents, which contain key-value collections (called "documents").
• Example: CouchDB, MongoDB, Riak

Graph Databases

• Lineage: Draws from Euler and graph theory.
• Data model: Nodes & relationships, both which can hold key-value pairs
• Example: AllegroGraph, InfoGrid, Neo4j

Scalability focus

How then do these data models scale to size and complexity? Check out this slide from my presentation at nosql east: 

The exact positions in the picture above are obviously debatable but I think it serves to illustrate my point: the key value stores and BigTable clones of the world handle size really well. This is because they have data models that can easily be partitioned horizontally. Which is great for scale out of, for example, simple two-column data like a whole bunch of username/password pairs.

The drawback however is that by constraining themselves to simpler data models, they've pushed complexity up the stack. So if you have data with a non-trivial structure, then you have to compensate for a simple data model by adding more complex functionality in the upper layers. [4]

Document databases and graph databases, on the other hand, have opted for richer data models. This means that they have more powerful abstractions that make it easy to model both simple and complex domains. But these richer data models introduce more coupling of data and therefore it's more challenging to get them to scale to size.

Size matters (but you're not Google so complexity matters more)

Now, size gets a lot of attention because scaling out to hundreds of machines is very sexy. But here's the kicker: the majority of the use cases out there don't need to store hundreds of billions of objects and scale out to truckloads of machines.

At the end of the day, there are only so many projects of Amazon and Google scale out there. A lot of projects fit within a couple of BILLIONS of objects. For most people, it's a lot more important to have a rich data model that lends itself to easily represent their domain.

Ben Scofield of Viget Labs expresses it eloquently in NoSQL Misconceptions:

"... there's a lot more to NoSQL than just performance and scaling. Most importantly (for me, at least) is that NoSQL DBs often provide better substrates for modeling business domains. I've spent more than two years struggling to map just part of the comic book business onto MySQL, for instance, where something like a graph database would be a vastly better fit."

Choose your hammer wisely

It's important to note that these data models are all isomorphic. Which is a fancy way of saying that you can express all datasets in either one of them. For example, you can decompose any data into a collection of key-value pairs.

But that's a bit like claiming you can write any program in any Turing complete programming language: sure, it's true in theory but just because you can doesn't mean that you should. In practice there's a bunch of programming languages that are a poor fit for many use cases. And the same is true of data models.

I think it's clear that we're rapidly moving beyond the era of the One Size Fits All database. Whereas in the past you could always trust that any decent-sized app had a relational database as backend, it's now increasingly about matching your dataset to whatever data model fits best. NOSQL is not No To SQL. NOSQL means Not Only SQL, as in: in the future, our backends will consist of Not Only SQL databases but also key-value stores, graph databases and more.

NOSQL is about choice and picking the right tool for the job. When you look at adding a NOSQL database to your current project, consider your requirements both for scaling to size and for scaling to complexity.

1] Few developers care whether their RDBMS implementation uses hash joins or nested loop joins.

2] Scaling to size and scaling to complexity was introduced (at least to me) in O'Reilly's Beautiful Data by Toby Segaran and Jeff Hammerbacher. The graph of the various NOSQL data models was first visualized by my friend and colleague Peter Neubauer.

3] Cassandra is actually the first of the "second-generation" NOSQL databases and it combines the decentralized scale out architecture of the Dynamo clones with the data model of BigTable.

4] As an analogy, imagine writing any piece of software and the only construct you had for storing state was a single global hashtable. No linked lists, no arrays, no structs, no objects. Imagine how much code you'd have to add just to work around that hashtable! Now, a key-value store is basically a distributed hashtable. This is why they have problems with scaling to complexity.