Implementing ServiceGraphPattern

[marqh]

Hello RDFLib

In trying to re-implement a system using rdflib I am facing a key capability gap: I require the ability to run queries that have SERVICE calls within the Query: federated queries, as defined by W3 in:

https://www.w3.org/TR/sparql11-federated-query/

Using rdflib (v4.2.2) I receive
Exception: ServiceGraphPattern not implemented
errors, from

rdflib/rdflib/plugins/sparql/evaluate.py

Line 262 in 58d6369

elif part.name == 'ServiceGraphPattern':

#484 suggests that this has been the case for some time.

I am interested in looking to implement this capability and I would like to start a discussion on how amenable RDFLib developers would be to an enhancement in this space?

I would also very much appreciate some advice on how to approach an implementation: what are the key factors to be aware of? what are the implementation and testing preferences within the developer community?

It strikes me that if it were very simple to do, it may have been done by now, so I would like to explore some of the potential complexities in principal before putting too much time into an implementation that will never fit with the RDFLib approach or simply never work.

I will add further comments on key details as I find them and explore implementation ideas with you as this ticket develops.

Your thoughts on whether this capability is of interest to RDFLib and how to implement it effectively are hugely welcome at this stage.

many thanks
mark

[marqh]

It appears to me there are 2 facets to this capability:

1. running the 'SERVICE' subquery remotely
2. providing the results into the main query

Regarding 2. the context: ctx is in scope at the point of identification of a SERVICE query fragment, so this should be achievable, although I'm not sure what form is required for this as yet.

However, there is something of an implementation challenge for 1.
At this point, there is a rdflib.plugins.sparql.parserutils.CompValue instance, a parsed representation of the SPARQL including significant rdflib implementation details. It is not obvious to me how to obtain the raw SPARQL query string from this.
If the raw SPARQL query string were available, a remote call to the relevant SPARQL endpoint could simply be implemented via a variety of means.

I feel that addressing this aspect is the key enabling factor for this work.

I am currently considering options including:

• passing the raw sparql fragment to the CompValue on creation, so that it can be retrieved and used later
• writing a regenerate_sparql method on CompValue to enable the sparql string to be regenerated on request

At present I feel like holding onto the raw fragment is less likely to introduce subtle bugs into the process. However, CompValue is an OrderedDict, so it is not easy to alter its __init__ parameters without changing behaviour, due to the **values as the last parameter to __init__.

Is there another way that the identified fragment of the SPARQL string that was parsed to create the ServiceGraphPattern CompValue could be in scope at the point of evaluate?
Could I look to subType CompValue to provide a CompValueWithSparql class that stores the SPARQL in raw form and enables it to be accessed directly within evaluate if the ServiceGraphPattern is identified?

Would this be able to be implemented within parser, say at

rdflib/rdflib/plugins/sparql/parser.py

Line 930 in 58d6369

ServiceGraphPattern = Comp('ServiceGraphPattern', Keyword(

Does parser have the raw string in scope at this point? (I'm still working my way through the logic of the parser module, I haven't grasped its functionality yet)

all assistance most gratefully received
mark

[gromgull]

Hi Mark!

It is indeed missing, I guess no one has really felt like they needed it previously.

We've had several use-cases where recovering the original text SPARQL query from the parsed query would be nice.

Ideally it would be reconstructed from the query-object, so that you could also modify that and get text out, but that's a bit more work.

As a hack, for prepareQuery calls we save the original string: https://github.com/RDFLib/rdflib/blob/master/rdflib/plugins/sparql/processor.py#L27

but you cannot guarantee that you query came in through that method.

[marqh]

Hi @gromgull

thank you for picking up on this.

I have had a few goes at splicing in some code to save an original string, but I keep coming up against the challenge that I only want part of the original string, the 'Service' clause. There doesn't seem to be a quick and dirty solution here.

I can't help thinking that the parsing to obtain parts is valuable and that what I want is a way to obtain a partial initial query, after the parsing has occurred.

We've had several use-cases where recovering the original text SPARQL query from the parsed query would be nice.

Ideally it would be reconstructed from the query-object, so that you could also modify that and get text out, but that's a bit more work.

I have looked into rdflib.plugins.sparql.parserutils and the Comp and CompValue classes.

Comp seems like it is more closely related to pyparsing.

I have also looked into
https://pythonhosted.org/pyparsing/pyparsing-module.html#originalTextFor

In think there might be a path here, but I can't quite get my head around the relationship between Comp and CompValue

By the time I reach the

    elif part.name == 'ServiceGraphPattern':
        raise Exception('ServiceGraphPattern not implemented')

part of rdflib.plugins.sparql.evaluate.evalPart I only have part in scope, which is a CompValue and has no relationship back to pyparsing that I can find.

Is there a way we could explore this option a bit more together. I feel there is a good solution here, but I can't fit the pieces together?

many thanks
mark

[marqh]

I also wrote a simple test case (though with no assertion as yet) to exercise these parts of the code:

test_sparql_service.py

from rdflib import Graph, URIRef, Literal
from rdflib.plugins.sparql import prepareQuery
from rdflib.compare import isomorphic


def test_service():
    g = Graph()
    q = '''select ?dbpProperty ?dbpValue
    where
    { service <http://DBpedia.org/sparql>
    { select ?dbpProperty ?dbpValue
    where
    {
    <http://dbpedia.org/resource/John_Lilburne> ?dbpProperty ?dbpValue .
    } } }'''
    g.query(q)


if __name__ == '__main__':
    import nose
    nose.main(defaultTest=__name__)

[marqh]

Hello @gromgull et al

I have finally made some time to try and progress this. I have got as far as obtaining the full text of the subquery and running it remotely via requests and the sparql end point.

There are some corners cut and some scope limits, so this is on a branch of mine at present, not a PR here:
marqh#1
I have added the expression to the Comp object
https://github.com/marqh/rdflib/pull/1/files#diff-7839f066926df815492a71be68544a7dR214
which enables a selection of the relevant part of the instring in postParse
https://github.com/marqh/rdflib/pull/1/files#diff-7839f066926df815492a71be68544a7dR225
for the service case

I can then pick this up and use it in evaluate
https://github.com/marqh/rdflib/pull/1/files#diff-8939ddb2af331e50406fdd945bf5fb2cR274
(with a little messing about via re for now)
to run the query remotely as required

However, I don't understand how to take the response I get and provide this in the return so it is in scope for the outer query
https://github.com/marqh/rdflib/pull/1/files#diff-8939ddb2af331e50406fdd945bf5fb2cR293

The test case
https://github.com/marqh/rdflib/pull/1/files#diff-451124cf0deee65995d1c871aaf86d53R6
has nothing in results.bindings

indeed, the first call to results.bindings, say at:
https://github.com/marqh/rdflib/pull/1/files#diff-451124cf0deee65995d1c871aaf86d53R22
returns
*** AttributeError: AttributeError("'<rdflib.plugins.sparql.processor.SPARQLResult object at 0x7f4968491210>' object has no attribute 'bindings'",)
the second call (straight after the first) returns
[]

so i'm clearly mis-understanding how to pass useful information out from this sub-query

some help at this stage would be very much appreciated. I am fumbling round in the dark somewhat

thank you
mark

[JervenBolleman]

@marqh I think I figured out how to get a binding fixed.
https://github.com/JervenBolleman/rdflib/tree/federate

We need to build a FrozenBinding which is something like this.

I ran into the same issue that RDFLib does not support SERVICE and would really like it, so continued your work. If you are still interested lets see if we can get this into a mergeable state together.

Fixed in #943