Emil's Neo Thoughts

NOSQL, The Web And The Enterprise

I can’t believe it’s already been two years since we raised our seed round! Oct 2009 saw a nascent NOSQL movement and Neo Technology as a two man band in Malmö, Sweden. Today, NOSQL is exploding and Neo has grown to a 25 person orchestra across two continents and five countries.

During these two years we've released  Neo4j 1.0 (after 10 years of development!), coined the NOSQL = Not Only SQL expansion at nosql east, heard the CTO of Amazon proclaim that “Neo4j absolutely ROCKS,” watched Facebook tell the world that it’s all about graphs, co-founded the Spring Data project to provide excellent support for NOSQL in the world’s most popular Enterprise Java middleware, changed our open source licensing to enable graph database ubiquity, made several kickass releases and started putting graph databases in the cloud.

But all of that is dwarfed by the amazing things our customers and community have done with Neo4j! Neo4j downloads grew by 10x last year and this year our growth has accelerated even more. Neo4j is clearly taking off.

The Evolution Of NOSQL

In parallel, it’s been fascinating to see the evolution of NOSQL these last two years. NOSQL is exploding. But in at least one aspect, it is falling short of its potential. Today, NOSQL is by and large a Web phenomenon, not an enterprise success story.

Sure, there’s some uptake in the enterprise. But that's exclusively in peripheral and non-critical applications like storage systems for logs, and never for mission-critical functionality like backing revenue-impacting always-on systems.

Why is that? The enterprise certainly sees the same explosion in data complexity and volume as the web world. But it turns out that very few NOSQL databases can meet the demands of the enterprise. Spearheaded by companies like Amazon, Google and Facebook most NOSQL databases are designed for the requirements of high-volume web properties, which aren’t necessarily the same as the enterprise requirements.

This prompts the question: are there things missing from today's NOSQL databases that the enterprise needs and demands?

Enterprise Expectations

We've talked to developers, architects and CxOs at many of the biggest enterprises out there about their database needs and three things stand out:

1. Support for transactions. Enterprise developers want to be able to group operations and have all of them succeed or none. This has been a basic and fundamental feature of all enterprise-strength relational databases since the early days and SQL wouldn’t be where it is today without it. Unfortunately, most NOSQL databases choose to relax consistency for availability. (For the right reasons, it turns out... well, at least if you’re Google!)
2. Support for durability. When you ask your database to store stuff, you want it to be there. Period. Unfortunately, most NOSQL choose performance over durability.
3. Support for Java. The enterprise application developer lives and breathes Java (and if they use something else, it’s not Ruby or PHP - it’s .NET). Unfortunately, most NOSQL databases come out of the web world where Java is the last thing they want to work with. Most have embryos of Java support, but none of it is great.

NOSQL For The Enterprise

Now, Neo4j is very different from other NOSQL databases in several ways. The first thing that comes to mind is its powerful graph data model, which is a superset of all the document and key value data models. That’s the fundamental basis upon which everything else is built. But how does Neo4j stack up against the above mentioned enterprise criteria?

1. Support for transactions: We wrote our first JTA/JTS-compliant logging transaction manager in 2002 and have supported full ACID-style transactions (including XA-compliant distributed two-phase commits) in production since 2003. Check!
2. Support for durability: This one was obvious to us. When we first started working on Neo4j in 2000 it was unthinkable to have a database that didn’t store stuff. Our opinions in 2011 haven’t changed. When you hit commit in Neo4j, your data will never get lost. Ever. Check!
3. Support for Java: Well, the 4j in Neo4j is actually “for the JVM.” Neo4j has excellent bindings to other languages like Ruby, Python and PHP. But we’ve always been Java first. We live and breathe Java every hour of every day. Check!

The enterprise sees the same explosion in data complexity and volume as the web world. And Neo4j is a perfect match for handling that data.

Building The Enterprise NOSQL Company

At Neo Technology we’re building the enterprise database company for the post-relational era, serving companies across the globe with solutions for managing complex and connected data. That is a major undertaking and a huge aspiration. And it is why today we are announcing that we’ve raised $10.6M in a Series A financing. This investment was led by Fidelity Growth Partners Europe with participation from existing investors Sunstone and Conor. Additionally, open source and enterprise software veteran Rod Johnson (who created the Spring framework which now has three million active enterprise developers) participated in the funding and will join our board as chairman.

Today marks an important milestone for Neo Technology, the Neo4j community, our customers and - I believe - for NOSQL in the enterprise. It is early still, but the database landscape is ripe for a disruption of schumpeterian proportions. Two years ago NOSQL was a niche and contrarian hacker movement. Today it's established in the web world. Two years from now it will be broadly adopted across the enterprise. This funding allows Neo Technology and the Neo4j community to fully participate in (and contribute to!) that historical transformation.

Graph Databases, Licensing and MySQL

Today marks a new major milestone for Neo4j: we’re making the core graph database - Neo4j Community - available under the same proven open source license as MySQL, the GNU General Public License (GPL).

That means that in every scenario where you can use MySQL for free, you can now also use Neo4j Community for free.

Why Is This a Big Deal?

This is a big deal because the world has lacked a great graph database with no barriers to adoption. The last year has seen tremendous interest in graph databases. We've seen a rapidly growing Neo4j community, an explosion in customer deployments, and many new graph database implementations in the last 12 months.

But we've learnt that the opportunity is still bigger. Some developers found the Neo4j license too restrictive, preventing their teams from using Neo4j. We listened, and that’s what we are changing today.

The graph database opportunity is at least as big as the MySQL opportunity. The best way to make graph databases achieve their full potential is to make a robust and mature graph database widely available under the same license as MySQL.

Wow! Why Are You Doing This?

It’s been almost two years since the first NOSQL conference in San Francisco in June of 2009. These two years have seen an explosion of NOSQL databases and a surge in interest, evaluation, experimentation and usage by everyone from scrappy startups to global enterprises.

Since the graph data model is the richest of all the NOSQL data models, graph databases have the opportunity to be the most useful for most people in most situations, and ultimately grab the largest slice of the NOSQL market.

Most users today store the vast majority of their transactional information in a relational database, with only a few using a NOSQL database:

Database use in most deployments 2011

But when we talk to some of the largest web properties in the world, the situation is more like this:

Database use in high-end deployments 2011

While the Amazon's and the Yahoo!’s of the world remain significant SQL users, NOSQL is also central to their business. Why? Certainly not because they enjoy writing their own databases! They were forced to deal with information of such size and complexity that they simply could not efficiently manage it all in a relational database. Hence they had to build their own custom in-house NOSQL datastores to satisfy business requirements.

Information is growing today at staggering rates in both size and complexity. Last year more new digital information was generated than during all previous years... combined! This information is highly connected, messy and complex. Nonetheless, it contains vast amounts of business value.

Businesses will want to look at this information, analyze it, and make decisions on it – more and more frequently in real time. Over time, this explosion of complex information will force all deployments to deal with increasingly larger volumes and increasingly higher information complexity. This will eventually impact all database-backed deployment out there and they will grow more and more similar to today's high-end deployments: using both SQL and NOSQL equally.

Database use in most deployments over time

Conclusion

Since the graph data model is the richest of all the NOSQL data models, graph databases have the opportunity to be the most useful for most people in most situations, and ultimately grab the largest slice of the NOSQL market.

In other words, graph databases have the opportunity to be ubiquitous. And because of the MySQL-like scale of this opportunity, we feel that the world deserves a robust and mature graph database licensed the same way MySQL is - under the GPL.

Therefore I'm happy to announce the immediate availability of Neo4j Community under the GPLv3. For more information about the new 1.3 release (including huge store support and a new web UI) please see the announcement. For more details about the new licensing, see here.

And finally, feel free to download and check out the leading graph database in the world, Neo4j Community, now available under the GPL.

On The Facebook Open Graph and Graph Databases

Last week Facebook announced their vision of the Open Graph -- what they call "a more social web." Now that the dust has settled, it stands clear that if Facebook can pull this off it will be a transformative moment for the web.

There are many perspectives on Facebook's move[1] but what makes me most excited is that the announcement was ALL about graphs. From the Facebook Graph API to the Open Graph Protocol, it seems like you can't touch any part of Facebook without getting infected by graphs.

That's ok by me! And since I happen to dabble in graphs for a living, I thought I'd spend some time today picking apart the announcement and talk about how it relates to our corner of the world: graph databases.

Ait, let's get started: what is this Open Graph anyway?

The Open Graph is Facebook's name for all content on the web that "hooks into" the social graph. Examples include people, places, businesses and events. This is nothing more than the regular web content we've always used and loved (for example, a movie listing on IMDB) but it's annotated with structured information so that it can be related to other things in the social graph.

Ok, so what is the Facebook Graph API?

The Facebook Graph API is the interface through which an external software developer interacts with the Facebook service. It's an HTTP API[2] that produces and consumes Open Graph data, i.e. people, places, businesses and the connections between them.

Whenever you as a web developer want to have access to data inside Facebook (say, if you'd like to figure out the friend graph of a visitor), you access that data through the Graph API. Facebook is basically a huge, highly connected graph and the Graph API is the lens through which we mere mortals can view and manipulate it.

Fair enough, then finally what is the Open Graph Protocol?

The Open Graph Protocol is a standard way for website authors to tag their content as part of the Open Graph. Those web pages can then be consumed not only by web browsers (for rendering) but also by services like the social plugins and Facebook (for processing).

Example: Let's say you have a website for tracking events such as concerts and book readings. The front page of the site is a chronological list of events. The markup for the front page consists of two things: the content (text saying "Dave Matthews Band") and then the markup (HTML tags and CSS elements saying "blue" and "font size x" or what not).

But there's a Band type in the current iteration of the Open Graph Protocol[3]. So what if you in addition to markup for presentation also added markup to say that the label "Dave Matthews Band" actually refers to a specific band in the Open Graph. Then you can add a social plugin, for example the "Like" button, to your front page, and that plugin can all of the sudden make sense of the content on your site.

When a visitor X to your site clicks "Like," the plugin knows to find the Open Graph content and throw that over the wire to the Facebook service. Suddenly all the friends of X see that X recommends checking out the upcoming Dave Matthews Band concert. Additionally, since your site has access to both the identity and location of X, you of course choose to highlight the events that are local to X as well as rank the events by popularity in amongst X's social graph.

If the Open Graph Protocol catches on it will create an unparalleled platform of machine-processable web content. The vehicle for inducing this change is an ecosystem of social plugins that will give website developers a shitload of cool functionality -- if only they sprinkle some Open Graph markup in their pages. Facebook's bet is that these plugins are appealing enough to push site owners over the (admittedly fairly low) threshold to tag their content with Open Graph markup.

Ok, so what's the bigger story here?

After taking all this in, two high-order bits stand out:

1. Facebook is expanding the notion of the social graph to include not only people and their friends but everything of value in the digital world.
2. Facebook is determined to capture how things are connected in that graph. Who is friends with whom? How are they friends? Who likes what? How much do they like it? All those questions are about relationships in the graph. Facebook believes they can derive great value from owning the answers to those questions.

This view of the web and of the world is important because now that we enter the age of Facebook, interacting with the Graph API and the Open Graph will be a common task. It will be as natural to the modern developer as storing shopping carts in MySQL tables was to the developer of the previous decade.

That doesn't mean MySQL will be displaced or that the Graph API even solves the same problem. It doesn't. But an entire generation of software developers will grow up and think natively in graphs. And that's a big deal.

How do graph databases and Neo4j relate to all this?

Graph databases are increasingly relevant in a world that's increasingly connected. Facebook's recent move is undeniably a significant step in that direction. 

But that's obvious.

So let me dig a little deeper and take this perspective: Facebook seems to think (and I agree with) that in this decade a lot of the value in the web ecosystem will reside in the Open Graph layer. A substantial amount of that value will be captured by Facebook, much in the same way that Google through pagerank captured the value of the link graph of the previous decade.

Image shamelessly stolen from Chris Messina who wrote an excellent post on the Open Graph Protocol, image based on original work by martiniko.

Now, some people will be satisfied with letting Facebook store the data they add on top of the Open Graph. They trade the value of that data for the convenience of social plugins and for not having to build and run large data centers and write their own large-scale graph databases. That's a painful investment Facebook has already been forced to make.

But others clearly won't want to donate that precious information to Facebook. They want to store parts of the Open Graph locally and they want to augment it with their own data, because that's the value of their service.

Of course, that data -- either being directly graph-oriented or operating on graphs -- fits extremely well in a graph database. Why take the pain of squeezing a graph into a table? Especially when the reward for all your hard graph/table mapping work is that you can barely even analyze it! Generally speaking, a relational database breaks down (due to joins) when you try to traverse more than 2-3 hops in any non-trivial connected data set.

Instead, I think the trend will continue where developers migrate towards a data model that fits the natural form of the data they're working with. In this case a graph database. Especially if that graph database is a robust and mature technology like Neo4j with years of solid 24/7 production track record and performance numbers on a level where it can traverse 1-2 MILLION hops per second in a graph.

Oh and wouldn't it also be neat if you could interact with all that data in your local graph and the Facebook graph in a uniform manner? Say, through the same API? Why, that'd be a neat addition to a graph database...

Conclusion

In conclusion, this announcement reinforces the fact that we're moving to a highly connected world. A world where the value lies in how things are connected. This megatrend has so far been mostly visible in the social web space.

But Facebook's gauntlet has been thrown: the Open Graph connects everything of value in the digital world. As Facebook's march towards dominance begins, developers around the world take notice that the natural and most powerful data model for anything web, really anything at all, is a graph.

It's a good time to be in the graph database space.

1] Not the least which is the implications it has for privacy and centralization of power, of course. But I'll try to stand clear of that or it'll take another ten pages before I get to how this all relates to graph databases.

2] The API uses HTTP but it's not fully RESTful. For example, it chooses to expose the identifiers of the connections and then the client use a predefined URI scheme to figure out the final URI of the connection. A RESTful API would use hypermedia to directly link the connection URI.

3] Astute readers will observe that there's a gap here. Who defines these types? Top-down or bottom-up? If bottom-up (as indicated in the spec), how do you define semantic equivalence? There's a lot unsolved (or unrevealed) parts about how all this will be governed.

oOo

NOSQL: scaling to size and scaling to complexity

About a week ago, following nosql east in Atlanta, Jonathan Ellis from the Cassandra project published a fantastic overview of the current NOSQL ecosystem. He analyzes 10 popular NOSQL databases along three axes: horizontal scalability, data model and internal persistence design. It's a great read.

The third axis (internal persistence design) may not be terribly relevant for users of NOSQL systems [1] but the position on the first two axes reveal some important underlying assumptions. In particular, it reveals a focus: is this NOSQL project oriented around scaling to size or scaling to complexity? [2]

The four main NOSQL data models

Now, there are four main categories of NOSQL databases today. Before we get into how they differ in focus, let me just quickly run through them and outline a few key characteristics:

Key-Value Stores

• Lineage: Amazon's Dynamo paper and Distributed HashTables.
• Data model: A global collection of key-value pairs.
• Example: Voldemort, Dynomite, Tokyo Cabinet

BigTable Clones (aka "ColumnFamily")

• Lineage: Google's BigTable paper.
• Data model: Column family, i.e. a tabular model where each row at least in theory can have an individual configuration of columns.
• Example: HBase, Hypertable, Cassandra [3]

Document Databases

• Lineage: Inspired by Lotus Notes.
• Data model: Collections of documents, which contain key-value collections (called "documents").
• Example: CouchDB, MongoDB, Riak

Graph Databases

• Lineage: Draws from Euler and graph theory.
• Data model: Nodes & relationships, both which can hold key-value pairs
• Example: AllegroGraph, InfoGrid, Neo4j

Scalability focus

How then do these data models scale to size and complexity? Check out this slide from my presentation at nosql east: 

The exact positions in the picture above are obviously debatable but I think it serves to illustrate my point: the key value stores and BigTable clones of the world handle size really well. This is because they have data models that can easily be partitioned horizontally. Which is great for scale out of, for example, simple two-column data like a whole bunch of username/password pairs.

The drawback however is that by constraining themselves to simpler data models, they've pushed complexity up the stack. So if you have data with a non-trivial structure, then you have to compensate for a simple data model by adding more complex functionality in the upper layers. [4]

Document databases and graph databases, on the other hand, have opted for richer data models. This means that they have more powerful abstractions that make it easy to model both simple and complex domains. But these richer data models introduce more coupling of data and therefore it's more challenging to get them to scale to size.

Size matters (but you're not Google so complexity matters more)

Now, size gets a lot of attention because scaling out to hundreds of machines is very sexy. But here's the kicker: the majority of the use cases out there don't need to store hundreds of billions of objects and scale out to truckloads of machines.

At the end of the day, there are only so many projects of Amazon and Google scale out there. A lot of projects fit within a couple of BILLIONS of objects. For most people, it's a lot more important to have a rich data model that lends itself to easily represent their domain.

Ben Scofield of Viget Labs expresses it eloquently in NoSQL Misconceptions:

"... there's a lot more to NoSQL than just performance and scaling. Most importantly (for me, at least) is that NoSQL DBs often provide better substrates for modeling business domains. I've spent more than two years struggling to map just part of the comic book business onto MySQL, for instance, where something like a graph database would be a vastly better fit."

Choose your hammer wisely

It's important to note that these data models are all isomorphic. Which is a fancy way of saying that you can express all datasets in either one of them. For example, you can decompose any data into a collection of key-value pairs.

But that's a bit like claiming you can write any program in any Turing complete programming language: sure, it's true in theory but just because you can doesn't mean that you should. In practice there's a bunch of programming languages that are a poor fit for many use cases. And the same is true of data models.

I think it's clear that we're rapidly moving beyond the era of the One Size Fits All database. Whereas in the past you could always trust that any decent-sized app had a relational database as backend, it's now increasingly about matching your dataset to whatever data model fits best. NOSQL is not No To SQL. NOSQL means Not Only SQL, as in: in the future, our backends will consist of Not Only SQL databases but also key-value stores, graph databases and more.

NOSQL is about choice and picking the right tool for the job. When you look at adding a NOSQL database to your current project, consider your requirements both for scaling to size and for scaling to complexity.

1] Few developers care whether their RDBMS implementation uses hash joins or nested loop joins.

2] Scaling to size and scaling to complexity was introduced (at least to me) in O'Reilly's Beautiful Data by Toby Segaran and Jeff Hammerbacher. The graph of the various NOSQL data models was first visualized by my friend and colleague Peter Neubauer.

3] Cassandra is actually the first of the "second-generation" NOSQL databases and it combines the decentralized scale out architecture of the Dynamo clones with the data model of BigTable.

4] As an analogy, imagine writing any piece of software and the only construct you had for storing state was a single global hashtable. No linked lists, no arrays, no structs, no objects. Imagine how much code you'd have to add just to work around that hashtable! Now, a key-value store is basically a distributed hashtable. This is why they have problems with scaling to complexity.

Let's go

So TechCrunch spilled the beans with a nice writeup and the genie is officially out of the bottle: Two weeks ago today, we closed a $2.5M seed stage investment with kickass VC firms Sunstone Capital and Conor Venture Partners.

As our friends and family know, we have danced the wonderful VC dance for a while now. As a matter of fact, we met the Sunstone and Conor teams for the first time over a year ago. During this year we've dated regularly -- and happily so -- but have still not quite been ready to make that final leap from dating to marriage.

Until now. As the interest around the NoSQL space and alternative databases has just exploded over the past 3-4 months it's been increasingly clear to us that we want to and need to accelerate. We fully believe that we're moving towards what Ben Scofield of Viget Labs describes as a "pluralistic approach to storing our data," i.e. an architecture where a single application typically works with multiple databases, each storing the datasets they're best for.

In a world where it's all about choosing the database paradigm that best fits your dataset instead of squeezing everything into the relational database, in that world we believe there's a huge opportunity for graph databases. With Sunstone and Conor on board, this investment will give us the means to pursue it.