Possible extensions

[AlCap23]

Hey @MilesCranmer !

I've been thinking about possible extensions and wonder how you think about this:

1. Parametric Operations
   In short, I want to be able to supply function with hold tuneable constants, e.g. f(x, y , p) = x / ( y + p) where p is a constant with a tuneable value. This could maybe be done with predefined patterns of nodes, but also might be doable somewhat different?

2. N-ary Operations
   This one seems a little harder, but might be doable as well ( major refactoring ). My reasoning for allowing this is basically steering more into the field of program synthesis and allow chunks of expressions to be parsed. Possible applications for this might be inference of systems of equations based on "building blocks" with contain typical patterns. I know that in theory this is also possible using just binary ops, but the chance of discovery might increase based on the structural prior.

3. Arbitrary Expression Graphs
   This plays along with 2. and reuse of patterns is key here. In many (natural) systems, reoccurrence of features is common ( in classical mechanics this would be the sin(q) and cos(q) for describing the translational movement of an angular joint ). Within a classical binary tree, each expression needs to be sampled individually while in a DAG ( possibly "just" a topological sorted DAG ), the influence of a given node could be extended beyond its direct parent.

I've made some prototypes using Symbolics / MTK for this, but it its rather slow ( given that I need to build the function for each candidate ).

Something of a parallel part:
I've been working on some alternatives to GA/GP to solve symbolic regression, partly motivated by the latest trend of using RNN to sample the graph and also related to MINLP. If you're interested we could have a chat about this :).

Cheers!

Edit Uh, I just noticed that this might all be doable quite easily given that you generate dispatches using the OperatorEnum! Nice.

[MilesCranmer]

Hey @AlCap23,
Thanks for reaching out!

1. I think this is already possible, it just would require 2. to allow for operators with multiple parameters. Note that in SymbolicRegression.jl, you can set arbitrary constraints, so I commonly set a constraint that a power law operator (^) only has a constant right child – which is basically a parameter.

2. This requires a bit of work but in theory its perfectly doable. Within DynamicExpressions.jl it shouldn't be too bad; it would get a bit harder in SymbolicRegression.jl because you need to change some of the formula involved in generating mutations/crossovers, etc. I've never had a use-case that justified the effort required to get this working though, as most operators can just be binary.

3. This is already possible, see Allow graph-based expressions instead of trees MilesCranmer/SymbolicRegression.jl#135 (now moved to this repo). When copying a tree, you would pass preserve_topology=true to preserve a graph structure.

Always interested in discussing alternatives to GPs, happy to chat whenever! I think it should be reasonably easy to just stick a learned sampling strategy here: https://github.com/MilesCranmer/SymbolicRegression.jl/blob/51d205c518eb3e99cfd45ac6a2d3dbbbd1944f32/src/Mutate.jl#L63. In general you don't want to do "too much" with neural nets inside the search process of a GP because they don't generalize well outside of a training set of expressions (it's simply too large of a space).

Cheers,
Miles

[AlCap23]

Thanks for the answer!

Awesome, I didn't know that 3. is already doable! I'll add you to a repo I've adjusted for DynamicExpressions yesterday, so maybe my intentions will be clearer ;) .

Additionally, the PR #15 could be useful. This is basically the ability to reuse certain parameter patterns within equation ( fix the number of unique parameters ). An example would be :

$$ f(x, p_1) = p_1 * x - \frac{x}{p_1 + x} $$

Where we want to add the constraint that all parameters are linked.

[MilesCranmer]

Thanks. For that use-case, I think the existing struct already works for it (without named nodes). Basically just have a single node and point to it from multiple parents => that parameter is thus linked.

[MilesCranmer]

Have a look at https://github.com/SymbolicML/DynamicExpressions.jl/blob/master/test/test_preserve_multiple_parents.jl for how this works

[AlCap23]

Hm, my test case was something like:

op = OperatorEnum(binary_operators = [-, *], unary_operators = [sin])
param = Node(; val = 3.0)
feat = Node(; feature = 1)
ex = sin(feat*param) - param
get_constants(ex) # Returns [3.0, 3.0], so each Node is still treated as an individual
set_constants(ex, [2.0]) # Error which makes sense

And I tried to constraint it by using the hash map, but this did not work out either. Another approach would be to just store the parameters and set them, which could also be done ( I think ).

[MilesCranmer]

Oh, for constants in particular, I actually didn't merge the update yet. But a working implementation is here: MilesCranmer/SymbolicRegression.jl@master...shared-node-utils#diff-dbc5c3e1c445f26c200c1ffef22b8451d9c6d30f2d02d51ff1a982e5789e39cfL127

[MilesCranmer]

I wonder if it could also be a good idea to have the Node type itself indicate whether nodes are to be shared or not. Then there would be no need for making an explicit indication that one wishes to copy topology or not.

[MilesCranmer]

Hey @AlCap23! I'm cleaning up some of the utilities for working with shared nodes: #17. I think it should be easier to add other functionalities now. I will try to keep the syntax for as much as we can preserve_topology=false as a keyword argument. Setting that to true will assume that shared nodes are in the tree.

[MilesCranmer]

@AlCap23 - Graph like expressions are working in v0.14!! 🎉

julia> operators = OperatorEnum(;
           binary_operators=[+, -, *], unary_operators=[cos, sin, exp]
       );

julia> x1, x2 = GraphNode(feature=1), GraphNode(feature=2)
(x1, x2)

julia> y = sin(x1) + 1.5
sin(x1) + 1.5

julia> z = exp(y) + y
exp(sin(x1) + 1.5) + {(sin(x1) + 1.5)}

Here, the curly braces {} indicate that the node
is shared by another (or more) parent node.

This means that we only need to change it once
to have changes propagate across the expression:

julia> y.r.val *= 0.9
1.35

julia> z
exp(sin(x1) + 1.35) + {(sin(x1) + 1.35)}

This also means there are fewer nodes to describe an expression:

julia> length(z)
6

julia> length(convert(Node, z))
10

where we have converted the GraphNode to a Node type,
which breaks shared connections into separate nodes.