Integrating mutable dependencies with redux

[tcoopman]

This is a question (not a bug) about how to integrate mutable dependencies into flux and redux.

I have a application that depends on mutable properties and I'm wondering how to integrate these with redux.

For example take a library for displaying maps (like openlayers, leaflet) and a following use case:

Use case:

• Display a map with geographical data
• Load a big intial data set (say 10 000 points) (DATA_LOADED)
• Have some actions like: LOCATION_ADDED, LOCATION_REMOVED, LOCATION_UPDATED
• Optional: Support undo, redo.

Example code

This is a very short example to show how the API of openlayers works to read json features.

var vectorSource = new ol.source.Vector({
  features: (new ol.format.GeoJSON()).readFeatures(geojsonObject)
});

var vectorLayer = new ol.layer.Vector({
  source: vectorSource,
  style: styleFunction
});

var map = new ol.Map({
  layers: [
    vectorLayer
  ],
  target: 'map',
});

vectorSource.addFeature(feature);

Ideal solution

If performance wasn't a problem I could implement the use case like this:

• Follow the redux architecture to the point. Have a store with

state: {
  features: [
    { 
      x: ...,
      y: ...,
      properties: {
      },
      ...
    }
  ]
}

and some reducers that update the features without any dependency on any API.

Create a component that that renders an openlayers map and receive the features that need to be rendered as props:

class Map extends React.Component {
    componentWillReceiveProps() {
        var vectorSource = new ol.source.Vector({
            features: (new ol.format.GeoJSON()).readFeatures(this.props.features)
        });

        var vectorLayer = new ol.layer.Vector({
            source: vectorSource,
        });

        // replace the vectorlayer on the map.
    }
    componentDidMount() {
        var vectorSource = new ol.source.Vector({
            features: (new ol.format.GeoJSON()).readFeatures(this.props.features)
        });

        var vectorLayer = new ol.layer.Vector({
            source: vectorSource,
        });

        var map = new ol.Map({
            layers: [vectorLayer],
            target: 'map',
        });
    }
    render() {
        return <div id = "map" / > ;
    }
}

So on every update of the props, we rerender completely

But

And this is a big but, this implementation is absolutely not acceptable for performance, even with a small number of features this becomes too slow very quickly!

So we need a better solution. Some ideas:

Possible solutions

Have the vectorSource in the state

This solves the performance issue, is easy to implement, but now I have a mutable state object and so I lose a lot of the immutable advantages.
Duplicating the vectorSource on every change, to keep the state immutable is also not a solution for performance and memory reasons.

Do a smart diff in the component

Instead of recreating the source in componentWillReceiveProps, do a smart diff on the previous and current props and do addFeatures, removeFeatures,...

This makes it possible to have immutable state again, but it adds complexity (diffing algorithm). If I can get the diffing good (so probably by giving the state the correct shape), the performance can probably be good enough.

Other solutions???

• Have a different mutable state object with the vectorSource separated from the immutable state. I'm not sure if this offers any advantages.
• ...?
• (Implement an Immutable vectorSource for openlayers... This could probably be a solution, but way too much work and out of scope)

Conclusion/Questions

At the moment I have implemented the first solution (vectorSource in the state) as this is the easiest, but I would like to migrate to redux and have a real immutable state. Also I don't like the coupling on the 3rd party API in my stores.

So I'm wondering if any of you have dealt with things like this:

• What do you think are good solutions to manage big data, mutable data with flux/redux?
• Do you see other patterns that can work. Am I missing a better solution?

Thanks

[eugene1g]

If generating vectorSource is prohibitively expensive, then immutability of that piece might be an unattainable ideal - user experience trumps engineering goals. Still, for predictability sake, perhaps something like this will work -

• Keep features and vectorSource in an isolated reducer to draw a boundary around the "messy" part - you know it's not perfect, but it does not contaminate the main app
• Sounds like for performance goals, incremental changes to features must be done in sync with reflecting those changes in vectorSource. Therefore this reducer will handle LOCATION_ADDED etc actions, and patch vectorSource at the same time as reflecting changes in features. The signature for handling source updates could still look like newVectorSource = applyChanges(currentVectorSource, changeData), even if applyChanges() returns the same vector object that is simply patched.
• Co-locating changes to the underlying features data with vectorSource information means that you have cheap incremental updates of vectorSource which meets performance requirements. This also means you do not need a complex diffing algo on the component level as the incoming vectorSource prop is always up to date, and your component can stick to rendering.
• Alternatively, if you are unsatisfied with having vendor-specific data (i.e. vectorSource) in the global app state, you could do defer vectorSource update and do it in the redux select() function. The problem with this is that you will need the diff algo to compare the structure of features every time. Perhaps you can make it cheap for cases when there are no changes by using ImmutableJS equality, but it will still be expensive if there are changes.
• Alternatively, you could write a custom redux middleware that listens for feature-changing actions and expands them to include domain-specific commands instructing how to update vectorSource. For example, if you call dispatch(setFeatureRating(1234,'Good')), then the middleware could expand the action to include _vectorPatch:[ {action:'update_color', nodeRef:1234, color:'green']}, {...} ). Then pass those commands to the component, and allow the component to do in-place patching of the source based on those commands.

[i4got10]

Waited a long time until someone will face the same problem with mutable dependencies, because I lack of English language skills to formulate the problem as it is.

Recreating the entire array of objects at each change of application state is terrible. This creates performance issues and creates frequent map redraws. At the moment I used this solution:

For each object on the map I created a react component, which is checked whether it should updated in shouldComponentUpdate. Each component stores a reference to the object on the map, and knows how to redraw it without recreating it from scratch.

class AreaMapZones extends React.Component {
    render() {
        // need to return something, it would be a set of empty noscript tags...
        return (
            <div>
                {this.props.zones.map(zone => <AreaMapZone zone={zone} map={this.props.map} key={zone.id}/>)}
            </div>
        );
    }
}

class AreaMapZone extends React.Component {
    render() {
        this._redraw();

        return null;
    }

    componentWillMount() {
        this._polygon = this._drawZone(this.props.zone);
        // add to map
    }

    componentWillUnmount() {
        this._polygon = null;
        // remove from map
    }

    shouldComponentUpdate(nextProps, nextState) {
        return !_.isEqual(this.props, nextProps);
    }

    _redraw() {
        // some work with this._polygon
    }
}

Not sure this solution is suitable for a really large number of objects, but in my case it solved the problem.

[gaearon]

@i4got10

Recreating the entire array of objects at each change of application state is terrible. This creates performance issues and creates frequent map redraws. At the moment I used this solution:

Have you tried using Immutable.js? It's much more efficient because it uses structural sharing.

[i4got10]

You can store your internal state in Immutable.js and it really efficient. But problem really is in another place.
For example, I am using YMaps, which produce a Map object with internal geoObjects collection. This collection is mutable, and its very handy to keep it in sync with your natural application state(which is immutable). You need to write some heavy diff algorithm, and apply some king of patches to that internal Map state.

function select(zones) {
    return {zones: state.zones};
}

class Map extends React.Component {
    componentWillReceiveProps(nextProps) {
        // what to do here?
        // on one side is this._map with its own this._map.geoObjects
        // on other side this.props.zones
    }
}

module.exports = connect(select)(Map);

Take a look at @tcoopman code in first post, may be its more clear

[gaearon]

OK, I was just not sure your post was also connected to the original post, and I was wondering if you're asking about a different issue.

[tcoopman]

@i4got10 At the moment I just keep the mutable objects in my state (Map and VectorSource), and I don't see a better solution.

To have an immutable state and good performance, your data libraries need to support immutable (Immutable.js for example) so that's a problem.

I am wondering though if it would be possible to implement an Immutable Proxy object like this.
It's just an idea, and something that probably wouldn't work in practice:

let immutableList = new ImmutableProxy(new mutableList([1,2,3]));
immutableList1 = immutableList.push(4);
> [1,2,3,4]
console.log(immutableList)
> [1,2,3]

I'm not sure how to implement this and if it has any benifits, but the ImmutableProxy could save all the inserts, updates, deletes and replay them on the mutableList.

So if you have done push(5), the ImmutableProxy saves it. When you request a previous version, the ImmutableProxy deletes 5 from the mutableList.

Of course, an Implementation like this is very fragile and wouldn't work if you do this:

let immutableList = new ImmutableProxy(new mutableList([1,2,3]));
immutableList1 = immutableList.push(4);
immutableList2 = immutableList.remove(3);

So it's just an idea.

I will probably try to port my application to redux and keep the mutable state as separated as possible (and probably duplicated sometimes) from the immutable state and hope for the best.

Of course, you lose the ability to use redux devtools. @gaearon would it be an option to add external hooks in devtools that let you reset your mutable state to the state requested? The application would then be responsible for restoring/rehydrating the mutable state.

[heyimalex]

Re: Do a smart diff in the component

This issue looks like a variation on something I've seen a few times in React; how do you change an imperative api to be declarative. Just chiming in to say I've had plenty of success doing diffing manually. It's tedious to write the code, and if the api you're bridging is enormous it can be very hard to get right because of interactions between changes, but it works.

The deciding factor for me is the surface area of the api. For something like this where you have three actions and little interaction between them, it's not super onerous to build.

[gaearon]

Closing, as it's been inactive for some time, and there's nothing actionable for Redux team here.
Feel free to continue discussing!

[tcoopman]

I just want to add a how I've solved this at the moment.

All my mutable state goes in one part of the state tree.

state = {
  mutable: {},
};

For my specific use case, openlayers actually handles a part of the UI (the map). Features get added for example without going through React.
I found a way to handle this:

I add a sideEffect function to the action:

export const positionFetch = (url) => (sideEffect) => (time) => {
  return {
    types: types.POSITION_RECEIVED,
    payload: {
      //....
    },
    meta: {
      sideEffect,
    },
  };
};

I've patched react-redux to add the state to mapDispatchToProps. (see reduxjs/react-redux#93). Not strictly necessary, but it works nice in my case.

// source must be an openlayers source http://openlayers.org/en/v3.8.2/apidoc/ol.source.Source.html
function addFeature(source) {
  return (feature) => {
    source.addFeature(geo.readFeature(feature));
  };
}

function selectActionCreators(dispatch, state) {
  return bindActionCreators({
    positionFetch: positionFetch(insConfig.url)(addFeature(state.map.services.get('positions').get('layer').getSource())),
  }, dispatch);
}

export default connect(
  selectState,
  selectActionCreators,
)(ConfigView);

The reducer updates the state, but also executes the sideEffect with the payload:

export function position(state = initialState, {type, payload, meta}) {
  switch (type) {
  case types.POSITION_RECEIVED:
    meta.sideEffect(payload);
    return state.update('positions', p => p.push(Immutable.fromJS(payload)));
  default:
    return state;
  }
}

I could write some middleware that executes the sideEffect, but for now I do it in the reducer, because that's also easier to only do it after a promise has succeeded, but it would of course be perfectly possible (and maybe I will change that later).

At the moment I find this solution good enough. It works and it is pretty flexible.
If you see things to improve this, please let me know.

[Robert-W]

@tcoopman Do you have any example repo's demonstrating the full workflow. I have some similar problems to yours but I am using ArcGIS JavaScript API with webmaps created in ArcGIS Online. Where I work we have tried to avoid putting the map in the store since it is mutable, but there are too many smaller pieces of the map to keep track of to put in the store (like graphics, rendering rules, active features in info windows, highlights, location, etc. there is still a lot more in a complex app), so currently it is in the store. Plus that feels like it is taking us away from a single source of truth since all that information already exists in the map. Have you found a good way to deal with the map?

Currently we have it in the store and leave it alone, and then just emit change events when the map event system fires an event so our components can grab data from the map. When the user interacts with the UI, we fire off an API call after the store updates from the components (e.g. checkbox gets toggled and when the props come back in saying its on or off, we hide/show the corresponding layer). But this makes the store not serializable, not to mention can cause issues with undo/redo or resetting to default state.

We also tried putting it in the global scope and duplicating a little data to help render the UI, but my colleagues hate the idea of globals (and I'm not a huge fan either). Then there is still the case where we need to fire change events when the map emits events, like infoWindow selection change or some other events fired from map interactions.

Seems like all the solutions we have tried out there have at least one caveat if not more.

[VictorQueiroz]

What if you need to update something like a RTCPeerConnection instance from an action? There's a better approach for that?