Singleton operator

[xogeny]

When I say "singleton" operator, I'm referring to the [] without a predicate expression inside that is seen in several examples (for example, this one).

If you look at that example, the expression in question is Phone[][type="mobile"].number and the expected result is ["077 7700 1234"] if the operator is present (and "077 7700 1234" without).

One thing to note about this operator is that it is relatively independent of position. For example, the following expressions all return the same expected result:

Phone[type="mobile"][].number
Phone[type="mobile"].number[]
$[].Phone[type="mobile"].number

This is problematic because the evaluation semantics of this operator are not like any other operators. For operators such as + or & we have a binary operator node and that node has operands that the operator applies to. We really don't have that with []. Instead, the [] operator modifies certain properties of its parent node and these modifications are independent of what type of node it is.

The reason I see this as a problem can be demonstrated by the last example expression Phone[type="mobile"].number[]. Note that the [] suppresses flattening of results. But it seems very odd that it should have the same impact if it appears after number than if it appears after Phone. There are several reasons this is odd. First, is that while processing a path an array is potentially created at each step of the process. A normal user probably assumes that the placement of the array has something to do with which arrays it applies to. But it turns out that it actually applies to the result of the whole path since it marks the result of the path expression and that marking is only considered at the end of processing a path or a name).

Another reason, by the way, for seeing this as a problem has to do with my old friend the block node. 😄. I say this because (Phone[type="mobile"].number)[] means that the [] is ignored (since it isn't being applied to a path. So it needs to be clearer that it is the path that this operator is impacting and that a block is actually interfering with the precedence of the operator (it can no longer be bound to a path if we inject the block).

So a few questions.

• It seems that we can consider this to be effectively impacting (exclusively) the evaluation of a path. Is this correct? I don't think I'm missing anything, but I just want to make sure.
• If so, should it be a semantic error to apply [] to a non-path? Currently, it isn't. The exerciser will accept an expression like (Phone[type="mobile"].number)[] or {"one": [1]}[] without issue even though the [] is meaningless in these contexts.
• If so, wouldn't it be better to use a syntax that makes it clear it is position independent? I don't actually care about the syntax, but it seems like using it as a unary operation on the path is the correct way to consider its evaluation. So something like []Phone[type="mobile"].number or Phone[type="mobile"].number[] (exclusively at the end) does a better job of conveying the scope of the operator. If implemented in such a way, then you could use the presence of the operator to properly mark the path/name expression as preserving singleton results.

Comments?

[michaelthoward]

Some comments from a JSONata newbie ...

• I have struggled mightily to get my head around the array vs. auto-flattening-sequence-like behavior of JSONata ... as evidenced by my questions on stackoverflow.
• I still don't fully understand it and am surprised when results are/are-not flattened.
• with that caveat, I am going to "go out on a limb" and give you my thoughts

A normal user probably assumes that the placement of the array has something to do with which arrays it applies to.

I agree completely.

should it be a semantic error to apply [] to a non-path?

JSONata is generally quite lenient ... so if "semantic error" means "throw an exception" then I vote no.

I'm not sure exactly what you mean by non-path.
(The JSONata model I currently have in my head is that a query string in JSONata has path-like syntax, but the dots '.' are deceiving because they actually are a $map() + $reduce() operation. In that sense, nothing is a path.)

I think it makes sense to apply unary [] singleton operator to the results of a function which returns an array, so in that sense it makes sense to apply [] to a non-path.

But, it seems to me that if [] is applied to a non-array value then it should not affect the results.

my old friend the block node. 😄

To add to the discussion/confusion ...
Phone[][type="mobile"].number http://try.jsonata.org/H13GvU5iM
and
(Phone[][type="mobile"]).number http://try.jsonata.org/SkqYPI9sG

give different results because of the introduction of the seemingly innocuous block node.

Once the block is introduced, then it seems nothing can be done to the left of the dot '.' operator to get the array back. see http://try.jsonata.org/BJ_CZv5iG and http://try.jsonata.org/HykzQDciG. However adding the [] on the right of the dot '.' operator will successfully wrap the singleton http://try.jsonata.org/rkMdLv5jG

If so, wouldn't it be better to use a syntax that makes it clear it is position independent?

I think it would be (or would have been) better to not allow it to be position-independent. From the assortment of queries above, I (currently) think that the preferred/clearest case is Phone[type="mobile"][].number ... the singleton array returned by the query is the thing that we want to remain an array.

My understanding is that the purpose of the unary [] singleton operator is to preserve singleton results ... i.e. to work-around the auto-flattening-sequence-like-behavior in the case where the result is an array with a single value. Therefore, it only makes sense to apply the unary [] to an array.

In summary:

• unary [] singleton operator should be a postfix operator
• it should only have effect on its prefix array argument
• therefore it should not be position-independent
• the array may result from a path or an expression
• to be lenient, the unary [] singleton operator should have no effect in a non-array context

Again, as a JSONata newbie there may well be aspects of this that I am missing.

[xogeny]

@michaelthoward I'm afraid my original discussion of the issue tended to assume quite a bit of knowledge about the internals of JSONata. So let me try and address your comments and clarify things.

A normal user probably assumes that the placement of the array has something to do with which
arrays it applies to.

I agree completely.

Excellent. It is always good to hear from actual users when arguing that users will prefer something. :-)

should it be a semantic error to apply [] to a non-path?

JSONata is generally quite lenient ... so if "semantic error" means "throw an exception" then I vote no.

Actually, there are already semantic errors in JSONata, so we wouldn't violate any existing norm. When I say "semantic error" what I mean is that what they've asked for isn't meaningful or doesn't mean what they expect. Consider the JSONata expression "foo" * 2. It is syntactically valid, but it cannot be evaluated because it isn't meaningful (in JSONata) to multiple a string by a number. Similarly, the [] operator exists and it can appear anywhere a predicate can appear. But if it doesn't mean anything there (i.e., is a applied to a non path), it is ignored. The problem with being silent here is that people who use it in such a way have probably made a mistake and we could point this out. Note that there are cases where JSONata will simply return undefined and not thrown an exception. The question is, does this rise to the level of notifying the user of the misuse of the operator?

So my inclination is to make this a semantic error because if people apply [] to something that isn't an array, it probably demonstrates that they don't understand what the operator does. But ultimately this is @andrew-coleman's call. I'm just laying out my rationale.

I'm not sure exactly what you mean by non-path.
(The JSONata model I currently have in my head is that a query string in JSONata has path-like syntax, but the dots '.' are deceiving because they actually are a $map() + $reduce() operation. In that sense, nothing is a path.)

Well a path is a type of AST node in JSONata. It corresponds to either a name or a dot separated list of expressions. The point is that the lexer and parser form these path nodes, I'm just trying to evaluate them. You are absolutely correct that the . is deceiving and it really is a map operator (reduction generally only happens when you follow it with a ~> <function> expression).

I think it makes sense to apply unary [] singleton operator to the results of a function which returns an array, so in that sense it makes sense to apply [] to a non-path.

But at present, the [] does nothing for functions. So now you are talking about completely new evaluation semantics, FYI.

But, it seems to me that if [] is applied to a non-array value then it should not affect the results.

But again, the [] currently doesn't impact anything but a path. An argument could certainly be made that there should be a general argument to suppress flattening. But that isn't how [] is currently interpreted. Again, this is something for @andrew-coleman. But I would add that I've spent quite a bit of time poking around in the evaluation semantics and I share your concerns about many of these issues (e.g., the rules for flattening). I'd really like to evolve the current semantics into rules that are both easier to implement and more intuitive for users.

My understanding is that the purpose of the unary [] singleton operator is to preserve singleton results ... i.e. to work-around the auto-flattening-sequence-like-behavior in the case where the result is an array with a single value. Therefore, it only makes sense to apply the unary [] to an array.

Yes, but the thing to realize is that internally every step in a path is treated as an array (except the first one...sometimes). In other words, the . operator means "iterate over each element of this array and apply the following expression to each element". So you can't really just stick a [] in there because then you actually have to make the evaluation semantics more complicated by saying "iterate over each element of this array if it is an array and apply the following expression to each element. But otherwise, if it is a scalar value, then simply evaluate the expression of the next step using only that scalar value as input". This is partly why it actually only matters that the final value in the path evaluation is either flattened or not (which is why I think it is more intuitive to just put the [] at the end because that is where it really gets applied). Said another way, I don't see an actual use case for putting a [] in the middle of a path. Furthermore, if you start having both scalars and arrays flowing through a path, you are going to quickly find yourself in a kind of "quoting hell".

[michaelthoward]

Thanks for your prompt and thorough response.

the . is deceiving and it really is a map operator (reduction generally only happens when you follow it with a ~> expression)

I don't want to get side-tracked but ... the model I have developed is that there is always a $reduce($append) ... and that this is what accounts for the "flattening"

Well a path is a type of AST node in JSONata ... the lexer and parser form these path nodes,

Yes, I have seen ast_optimize()

But at present, the [] does nothing for functions. So now you are talking about completely new evaluation semantics, FYI.

Hmmm ... well I guess I didn't know that [] did nothing for functions. I wonder if that is also why putting [] around the block node has no impact.

internally every step in a path is treated as an array ...

Your explanation about the evaluation process seems very useful. I will need to chew on it a while in order to digest it.

Thanks

[xogeny]

internally every step in a path is treated as an array ...

Your explanation about the evaluation process seems very useful. I will need to chew on it a while in order to digest it.

Let me add a few more details. There is a "wrinkle" here which has to do with the first step. JSONata currently uses some heuristics to determine how to treat the first step. I'm hoping we can deprecate those heuristics but it is important to understand them if using v1.5.x.

Basically, if the first step in the path is an array or a variable, then it treats that as if it was the "input" value (i.e., it ignore the input value) and then each dot is a map operation. Otherwise, every step (including the first) is treated as a map operation. Again, I have ideas for streamlining this but I'm just letting you know how v1.5.x currently works.

[michaelthoward]

if the first step in the path is an array or a variable, then ... Otherwise, ...

Thanks for the warning.

it is important to understand them if using v1.5.x.

Unfortunately I must embed in Java 8 ... so I am currently stuck at ES5 jsonata-1.1.1.js

[xogeny]

@michaelthoward I'm a bit puzzled about what Java 8, ES5 and JSONata 1.1.1 have to do with each other. Is the issue that you are using a Javascript interpreter for Java 8 that only handles ES5? Isn't JSONata 1.5.x also available as ES5. What is the issue exactly?

Part of my interest is because I'm working on a (pretty big) PR for JSONata that involves moving JSONata to TypeScript and I'd like to make sure that I can support whatever your use case is. My version can target ES5 and ES6 (or even ES3). So I just want to figure out whether my version will support your use case...?

[michaelthoward]

Thanks for pointing this out ... it was my bad ... see below.

I haven't written any JavaScript in a decade (and didn't write much before that) ... so no doubt a large part of my problem is self-inflicted.

Is the issue that you are using a Javascript interpreter for Java 8 that only handles ES5?

Correct. The nashorn JavaScript engine in Java 8 is only ES5.

Isn't JSONata 1.5.x also available as ES5. What is the issue exactly?

In this thread @andrew-coleman says:

Since v1.2, jsonata.js uses ES6 features, so you'll need to make sure the version of Nashorn you are using supports this.

When I read this two weeks ago I understood it to say that I need to be at pre-v1.2 ... so I went with v1.1.1

Upon rereading it now I have a slightly better understanding of Andrew's comments.

*** some time later ****

I believe I have seen references to jsonata-es5.js ... however it is no longer listed in the source at
https://github.com/jsonata-js/jsonata

Q: Where-do-I-get/How-do-I-generate an es5 version of jsonata.js ?

[michaelthoward]

I learned enough about npm + babel to be able to generate jsonata-es5.js using current 1.5.3

However, Java 8 nashorn is croaking on require()

I have found several ways to make require() work in Java 8 nashorn http://source.coveo.com/2016/06/15/nashorn-require/
https://www.n-k.de/riding-the-nashorn/#_npm
https://github.com/walterhiggins/commonjs-modules-javax-script

Will advise

[raj-work]

Have you tried using J2V8 (https://github.com/eclipsesource/j2v8 <https://github.com/eclipsesource/j2v8>). We are using J2V8 and it helped us migrate to ES6 implementation in Java.

…

On Apr 12, 2018, at 10:10 PM, Michael Howard ***@***.***> wrote: I learned enough about npm + babel to be able to generate jsonata-es5.js using current 1.5.3 However, Java 8 nashorn is croaking on require() I have found several ways to make require() work in Java 8 nashorn http://source.coveo.com/2016/06/15/nashorn-require/ <http://source.coveo.com/2016/06/15/nashorn-require/> https://www.n-k.de/riding-the-nashorn/#_npm <https://www.n-k.de/riding-the-nashorn/#_npm> https://github.com/walterhiggins/commonjs-modules-javax-script <https://github.com/walterhiggins/commonjs-modules-javax-script> Will advise — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#196 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbTtwTpdc8Hklx2YC4OviaAhivlJnOTdks5toAkwgaJpZM4TNVvp>.

[michaelthoward]

Have you tried using J2V8

Thank you for the suggestion.

However, our my use case requires me to stay completely in the Java world, so JNI == Java Native Interface bindings to the C++ V8 engine isn't an option.

[michaelthoward]

I have spent this weekend trying to get jsonata-1.5.3-es5.js to load under the nashorn JavaScript engine in Java 8. Unfortunately, I have not been successful and need additional guidance.

The jsonata-es5.js which is generated by babel includes a number of calls to load modules via require() where the destination is a directory. Many of the node_modules subdirectories in the jsonata source have neither index.js nor package.json. For example the first thing referenced by jsonata-es5.js is

var _symbol = require('babel-runtime/core-js/symbol');

and this directory has neither package.json nor index.js.

I ventured a little bit down the path of using https://github.com/nodyn/jvm-npm as an alternate implementation for require(). I did not get very far because this implementation reads modules from the local filesystem ... and that is off-limits for my needs.

I switched to http://source.coveo.com/2016/06/15/nashorn-require/ because it is a Java implementation. This solves my filesystem problem because it gives one control over where to load the files from ... in my case from resources that are bundled inside a .jar file.

I pulled the entire node_modules subdirectory from the JSONata trunk.

However, as I said above, many of those subdirectories do not include index.js.
I have confirmed that the nashorn require plugin that I am using is attempting to open package.json and index.js on the subdirectory. Since that fails, all is failing.

I tried setting up node.js and jsonata from the command line on my machine to see if the node_modules that it pulled down would include directory package information, but I was unsuccessful ... just too much for me to try to get my head around in a few hours.

@xogeny wrote:

... TypeScript ... So I just want to figure out whether my version will support your use case...?

I don't know anything about TypeScript, so I don't know.
Using the babel-generated es5 version is proving to be difficult. If I were more familiar with the node.js world then perhaps it would easier.
Regardless, the babel es5 translation might not be practical for my use case. The entire tree of node_modules is auto-included by the translation, and I am hesitant to include it in my Java runtime.
The JSONata source itself is pretty clean, but the babel es5 translation pulls in a lot of hair.

Recommendations/advice would be appreciated ... thanks in advance.

[xogeny]

@michaelthoward I would suggest you open a separate issue. I have some comments, but I don't want to distract from the original issue here.

[andrew-coleman]

This is a great discussion and it's good to have feedback on the usability of certain constructs.

To address @xogeny's original questions:

• It seems that we can consider this to be effectively impacting (exclusively) the evaluation of a path. Is this correct? I don't think I'm missing anything, but I just want to make sure.

Yes, correct

• If so, should it be a semantic error to apply [] to a non-path? Currently, it isn't. The exerciser will accept an expression like (Phone[type="mobile"].number)[] or {"one": [1]}[] without issue even though the [] is meaningless in these contexts.

Hmm, or maybe the evaluator should apply the same semantics to non-path expressions as well. After all, non-path expressions (block expressions, function calls, etc) could return a result sequence whereby we might want to retain a singleton within an array.

• If so, wouldn't it be better to use a syntax that makes it clear it is position independent? I don't actually care about the syntax, but it seems like using it as a unary operation on the path is the correct way to consider its evaluation. So something like []Phone[type="mobile"].number or Phone[type="mobile"].number[] (exclusively at the end) does a better job of conveying the scope of the operator. If implemented in such a way, then you could use the presence of the operator to properly mark the path/name expression as preserving singleton results.

Probably. I struggled to think of an appropriate syntax that would clearly indicate this intention. I ended up overloading the filter [] syntax. I'm not particularly attached to it, so if anyone has a radically better idea then I'm open for a breaking change in v2.0.