Graph databases and RDF triplestores: storage of graph data in python

Question

I need to develop a graph database in python (I would enjoy if anybody can join me in the development. I already have a bit of code, but I would gladly discuss about it).

<

Answer 1

Hmm, maybe you should take a look at CubicWeb

Answer 2

Regarding Neo4j, did you notice the existing Python bindings? As for the disk storage, take a look at this thread on the mailing list.

For graphdbs in Python, the Hypergraph Database Management System project was recently started on SourceForge by Maurice Ling.

Answer 3

I have used both Jena, which is a Java framework, and Allegrograph (Lisp, Java, Python bindings). Jena has sister projects for storing graph data and has been around a long, long time. Allegrograph is quite good and has a free edition, I think I would suggest this cause it is easy to install, free, fast and you could be up and going in no time. The power you would get from learning a little RDF and SPARQL may very well be worth your while. If you know SQL already then you are off to a great start. Being able to query your graph using SPARQL would yield some great benefits to you. Serializing to RDF triples would be easy, and some of the file formats are super easy ( NT for instance ). I'll give an example. Lets say you have the following graph node-edge-node ids:

1 <- 2 -> 3
3 <- 4 -> 5

these are already subject predicate object form so just slap some URI notation on it, load it in the triple store and query at-will via SPARQL. Here it is in NT format:

<http://mycompany.com#1> <http://mycompany.com#2> <http://mycompany.com#3> .
<http://mycompany.com#3> <http://mycompany.com#4> <http://mycompany.com#5> .

Now query for all nodes two hops from node 1:

SELECT ?node
WHERE {
    <http://mycompany.com#1> ?p1 ?o1 .
    ?o1 ?p2 ?node .
}

This would of course yield <http://mycompany.com#5>.

Another candidate would be Mulgara, written in pure Java. Since you seem more interested in Python though I think you should take a look at Allegrograph first.

Answer 4

Redland (http://librdf.org) is probably the solution you're looking for. It has Python bindings too.

Answer 5

I think the solution really depends on exactly what it is you want to do with the graph once you have managed to store it on disk/in database, and this is a little unclear in your question. However, a couple of things you might wish to consider are:

• if you just want to persist the graph without using any of the features or properties you might expect from an rdbms solution (such as ACID), then how about just pickling the objects into a flat file? Very rudimentary, but like I say, depends on exactly what you want to achieve.
• ZODB is an object database for Python (a spin off from the Zope project I think). I can't say I've had much experience of it in a high performance environment, but bar a few restrictions does allow you to store Python objects natively.
• if you wish to pursue RDF, there is an RDF Alchemy project which might help to alleviate some of your concerns about converting from your graph to RDF structures and I think has Sesame as part of it's stack.

There are some other persistence tools detailed on the python site which may be of interest, however I spent quite a while looking into this area last year, and ultimately I found there wasn't a native Python solution that met my requirements.

The most success I had was using MySQL with a custom ORM and I posted a couple of relevant links in an answer to this question. Additionally, if you want to contribute to an RDBMS project, when I spoke to someone from Open Query about a Graph storage engine for MySQL them seemed interested in getting active participation in their project.

Sorry I can't give a more definitive answer, but I don't think there is one... If you do start developing your own implementation, I'd be interested to keep up-to-date with how you get on.

Answer 6

RDFLib is a python library that you can use. Using harschware's example:

Create a test.nt file like below:

<http://mycompany.com#1> <http://mycompany.com#2> <http://mycompany.com#3> .
<http://mycompany.com#3> <http://mycompany.com#4> <http://mycompany.com#5> .

To query for all nodes two hops from node 1 in RDFLib:

    from rdflib import Graph

    g = Graph()
    g.parse("test.nt", format="nt")

    qres = g.query(
        """SELECT ?node
        WHERE {
            <http://mycompany.com#1> ?p1 ?o1 .
            ?o1 ?p2 ?node .
        }"""
    )

    for row in qres:
        print(node)

Should return the answer <http://mycompany.com#5>.