Callback initialization with `None` vs default properties

[chriddyp]

This is a more thorough investigation of #396 and plotly/dash-renderer#81. plotly/dash-renderer#81 was a candidate for our 1.0.0 breaking change release and this issue is in part a response to that proposal.
As with all proposed breaking changes in Dash, this issue serves to provide enough historical and technical background to allow any of our community members to participate.

plotly/dash-renderer#81 is a proposal to change our Dash callbacks are fired upon initialization. In particular, it prevents the initial callbacks from firing if properties weren't explicitly provided.

Through this analysis, I've come to the conclusion that plotly/dash-renderer#81 isn't a complete solution to the underlying issues and inconsistencies. I propose a couple of alternative solutions towards the end. These solutions will require a larger engineering effort and design discussion and so may not be solved on the timeline of our 1.0.0 breaking change release.

Technical Background

Currently, when Dash apps load, Dash fires a certain set of callbacks:

• Callbacks with inputs that exist on the page
• Callbacks with inputs that aren't themselves outputs of any other callback.

If a callback's property wasn't defined in the initial app.layout, then it is supplied in the callback as None. If it was supplied, then it is provided in the callback as that value.

For example:

app.layout = html.Div([
    dcc.Input(id='input-1'),
    html.Div(id='output-1'),

    dcc.Input(id='input-2', value='initial value 2'),
    html.Div(id='output-2'),

    html.Div(id='input-3'),
    html.Div(id='output-3'),
    html.Div(id='output-3a'),
    html.Div(id='output-3b'),

    dcc.Input(id='input-a'),
    dcc.Input(id='input-b', value='initial value b'),
    html.Div(id='output-c')
])

@app.callback(Output('output-1', 'value'), [Input('input-1', 'value')])
def update_output_1(value):
    return 'output 1 update'


@app.callback(Output('output-2', 'value'), [Input('input-2', 'value')])
def update_output_2(value):
    return 'output 2 update'


@app.callback(Output('output-3', 'children'), [Input('output-2', 'children')])
def update_output_3(children):
    return 'output 3 update'


@app.callback(Output('output-3a', 'value'), [
    Input('output-1', 'children'),
    Input('input-1', 'value')])
def update_output_3a(children, value):
    return 'output 3a update'


@app.callback(Output('output-3b', 'value'), [
    Input('output-1', 'children'),
    Input('input-2', 'value')])
def update_output_3b(children, value):
    return 'output 3b update'


@app.callback(Output('output-c', 'value'), [
    Input('input-a', 'value'),
    Input('input-b', 'value')])
def update_output_c(input_a, input_b):
    return 'output c update'

In this example, Dash will:

• On page load, call:
  • update_output_1 with None
  • update_output_2 with 'initial value 2'
  • update_output_c with None and 'initial value 2'
• These callbacks will update the app and if any outputs are themselves inputs, then then this update will trigger another set of callbacks:
  • update_output_3 with 'output 2 update'
  • update_output_3a with 'output 1 update' and None
  • update_output_3b with 'output 1 update' and 'initial value 2'

The proposed change would change Dash so that callbacks with any inputs with properties that aren't supplied aren't called on initialization.

With this change, the following callbacks would be fired on page load:

• update_2 with 'initial value 2'

And after output-2 is updated, the following callbacks are triggered:

• update_output_3 with 'output 2 update'
• update_output_3b with 'output 1 update' and 'initial value 2'

Historical Context

Why are callbacks fired on page load?

Callbacks are fired on page load for consistency. Consider the following case:

app.layout = html.Div([
    dcc.Input(id='input', value='NYC'),
    dcc.Graph(id='graph')
])

@app.callback(Output('graph', 'figure'), [Input('input', 'value')])
def update_graph(value):
    return {'layout': {'title': value}}

In the current version of Dash, the figure property of the dcc.Graph goes through the following transitions:

1. Initialized and rendered as None
2. Dash fires off the page-load callbacks (update_graph), updating figure to {'layout': {'title': 'NYC'}}

After initialization, the app is in the same state on page load as it would be if the dcc.Input was empty and then the user wrote 'NYC'. In other words, the app's initializaiton state is no different than any other state in the application: the app is completely described by the values of the input properties and not by its history or previous state.

If we didn't fire the callbacks on page load, then the figure would remain as None but the input would be filled with 'NYC'. If the user deleted the text and then re-wrote 'NYC' in the input, then the graph would have {'layout': {'title': 'NYC'}}, which would appear odd as this is not what the graph looked like in the original state of the app even though the dcc.Input was in the same state. This is what we mean when we say that this is "inconsistent".

plotly/dash-renderer#81 proposes changing the behaviour when value in the dcc.Input isn't supplied. Consider this example:

app.layout = html.Div([
    dcc.Input(id='input'),
    dcc.Graph(id='graph')
])

@app.callback(Output('graph', 'figure'), [Input('input', 'value')])
def update_graph(value):
    return {'layout': {'title': value}}

plotly/dash-renderer#81 proposes that the update_graph callback should not be fired as the value property of the dcc.Input was not provided.

In the current version of Dash, if a property isn't explicitly supplied then it is passed into the callback as None. From the Dash user's perspective, it's the same as dcc.Input(id='input', value=None).

Inconsistencies with the existing behavior

Passing undefined properties as None into the callbacks is actually a little problematic:

1. Empty Value. In many components, None isn't actually the property's "empty" value as it's not a state that can be achieved through user interaction. Consider the following components:

   • dcc.Input(type='text') - The empty state for value is ''. That is, if the user deletes the text in the input box, '' is passed back to the callback, not None (type='number' may have a different "empty" value, perhaps None)
   • dcc.Dropdown(multi=True) - value is [] when there aren't any items in the dropdown, not None. However, if multi=False, then None is the valid empty state.

   This means that the Dash developer has to handle two different "empty" states of the property with logic like:

   @app.callback(Output('graph', 'figure'), [Input('input', 'value')])
   def update_graph(value):
       if value is None or value == '':
           return {'layout': {'title': 'No input specified, please fill in an input.'}}

2. None can be invalid. Since None isn't provided by the component, it isn't necessarily valid. This is non-intuitive because None was supplied to the callback from Dash and so it is assumed that it is the actual value of the property. This assumption would lead the user to believe that:

   dcc.Graph(figure=None)

   and

   dcc.Graph()

   would render the same result. They might, but it's not guaranteed. It's up to the component author to enforce this.

   From a component author's perspective, these two initializations are different. In the former, figure is supplied and its explicitly set to None. In the latter, figure isn't even supplied. The component author could render the component in different ways depending on which value was supplied. Handling these differences is most often done via defaultProps (see next point).

3. Default Value. When components are rendered in the front-end, the component author may provide default properties for the component (using e.g. React's standard defaultProps) if the properties aren't supplied.
   For example, consider dcc.Graph: the figure component could have the following states:

   1. If None, then don't draw anything, not even an empty graph.
   2. If not supplied, then set the default to be {'data': [], 'layout': {}}. This will draw an empty graph.

   In this case, dcc.Graph() would default to something reasonable (an empty graph) but if the user really wanted to clear the container they could set dcc.Graph(figure=None).

   However, if the user had a callback listening to the figure property, then they would receive None as an argument on initialization, which doesn't match what was rendered in the front-end ({'data': [], 'layout': {}}). dcc.Graph() would be rendered differently than dcc.Graph(figure=None) but in both cases None would be passed into the callback:

   @app.callback(Output(...), [Input('my-graph', 'figure')])
   def update_something(figure):
       # ...

   Similarly, consider n_clicks in html.Button. n_clicks represents the number of times that the element has been clicked and so it's intuitive that its default value is 0 (and it is). However, since Dash doesn't send this value to the callback, None is passed into the callback, causing a lot of confusion.

   app.layout = html.Div([
       html.Button(id='button'),
       html.Div(id='output')
   ])
   @app.callback(Output('output', 'children'), [Input('button', 'n_clicks')])
   def update_output(n_clicks):
       if n_clicks is None:  # users would rather write `if n_clicks == 0`
           return ''
       else:
           return do_something()

4. Computed Defaults. Some components have "computed" defaults. That is, their default properties can't be statically defined. For example:

   • dcc.Graph.id - Computed dynamically (a random string)
   • dcc.Location.path - The href property (among others in dcc.Location) isn't known until the component has rendered on the page as the user could be loading the page on any path, e.g. / or /some-page
   • dcc.Store.data - If the Store has data stored in local storage, then it must dynamically retrieve this data on page load.
   • dash_table.DataTable.derived_virtual_data - This property represents the data in a table after it has been filtered or sorted. Currently, it is not computed on initialization, but it should be. See the example here: https://dash.plot.ly/datatable/interactivity

   These dynamic defaults cause confusing initialization behavior. Consider the following example:

   app.layout = html.Div([
       dcc.Location(id='location')
       html.Div(id='some-div'),
   ])
   
   @app.callback(Output('some-div', 'children'), [Input('location', 'path')])
   def update_output(path):
       ...

   On page load, update_output is initially called with None and then immediately called with the actual path. In this case, None doesn't actually represent the "empty" state, it's more like the "unknown" state or the "uncomputed" state.

   So, to handle this use case, users should write their code as:

   @app.callback(Output('some-div', 'children'), [Input('location', 'path')])
   def update_output(path):
       if path is None:
           raise dash.exceptions.PreventDefault
       elif path == '/':
           # home screen
       elif path == '/page-1':
           # ...
       # ...

   With static defaults, the user could always remove the None checks by supplying their own initial value. For example, instead of dcc.Input() they could write dcc.Input(value=''). In the case of computed defaults, the user can't do this as they don't know what the properties will be. They can't write dcc.Location(path='/') because the the user won't necessarily land on / when they visit the page, they may land on /page-1.

Possible Solutions

Solution 1. Don't fire initial callbacks if inputs aren't supplied

• Change dash-renderer to skip firing the initial callbacks if the input properties aren't explicitly supplied
• Component authors could provide fire the callbacks when the component is rendered for the first time by calling setProps in the component lifecycle. This would allow components with computed defaults to fire their initial callbacks (e.g. dcc.Storage.data or dcc.Location.path)
• Properties supplied as State would be passed into callbacks as their default (computed or static) properties instead of None

Notes:

• By omitting an initial property value, users could prevent the initial request from occurring. Think of this as an "automatic", front-end version of raise dash.exceptions.PreventDefault.

• Components with dynamic properties would be responsible for firing the callbacks after initialization. This means that the "consistency" of whether or not callbacks are fired upon initialization is determined by the component author on a component-by-component basis.

• This could be confusing as dash devs would not know the initialization behaviour in advance. This may be difficult to explain to new users. Here's an example of what this documentation might look like:

  On page load, Dash will go through an application initialization routine where certain callbacks will be fired. In particular:

  • If you supply initial values to properties that are dash.dependencies.Input, then dash will fire your callbacks to compute the "initial outputs".
  • If your component's properties have "dynamic" defaults, then your callback may be fired with these dynamic, computed values. This varies on a component-by-component basis. Dynamic, computed component properties include:
    • All of the URL properties in dcc.Location (these are determined via the URL of the page when the component is loaded)
    • The derived properties in dash_table.DataTable
    • The data property of dcc.Store if type is localstorage
  • If you omit supplying a property value, and if that property value isn't computed dynamically by the component, then Dash will not fire your callback.

  Dash fires your callbacks on app start in order for your app to have consistent, default outputs. So, it is encouraged for you to supply explicit default input arguments. That is, dcc.Input(value='') instead of dcc.Input()

• This could improve initial page render performance time as fewer requests could be made.

• Users would be encouraged to supply initial values of their inputs so that the initial state of the app's lifecycle would be "consistent" and so that the outputs would have default values. The main exceptions to this rule would be n_clicks (users could omit this so that the callback wouldn't be fired until the user clicks on the button) and the computed properties (which can't be supplied by the user).

• This method would allow certain initial requests to be ignored but not all of them. This may be confusing to users: they may expect that they could programatically ignore an initial callback that has derived_virtual_data as an Input by just not supplying an initial value for that property. However, since it is computed, they can't ignore this callback.

• The default properties (computed or static) would need to be provided as State. In the following example, derived_virtual_data would be equal to data and figure would be something like {'data': [], 'layout': {}}

  app.layout = html.Div([
      dash_table.DataTable(id='datatable', data=df.to_dict('records')),
      dcc.Graph(id='graph'),
      dcc.Dropdown(
          id='dropdown',
          value='nyc',
          options=[{'label': 'NYC', 'value': 'nyc'}]
      ),
      html.Div(id='output')
  ])
  
  @app.callback(Output('output', 'children'),
                [Input('dropdown', 'value')],
                [State('datatable', 'derived_virtual_data'),
                 State('graph', 'figure')])
  def update_output(value, derived_virtual_data, figure):
      # ...

• If the property was supplied explicitly as None, the callback would still be fired. dcc.Dropdown(value=None) would fire the callback but dcc.Dropdown() would not fire the callback.

Solution 2. Fire callbacks with default (computed or static) properties

This solution would fire all of the callbacks on initialization but instead of passing in undefined properties as None, it would use the component's static or computed default properties.

Notes:

• If the property doesn't have a default value, it would be None. This would be the same as if the component had a default value and it was explicitly None.

  • Alternatively, if the component didn't have a default property, it could be set as a new property that we call "Undefined", closer matching the JavaScript API.

• The mechanism for retrieving the default properties from the component would be the same mechanism that "Solution 1" would use to retrieve the properties to populate State

• This behaviour is easier to explain:

  On page load, Dash will fire your callbacks. If you did not supply an initial value for a property that is an Input or State, Dash will pass in a default property.
  You can find the default properties by calling help on the component (e.g. help(dcc.Dropdown)) or viewing the component's reference table (e.g. https://dash.plot.ly/dash-core-components/dropdown). Some of these properties are dynamically computed (e.g. the URL properties of the dcc.Location component) and the documentation for the component should indicate this.

• Certain components frequently would have PreventDefault in their canonical usage, like html.Button:

  @app.callback(Output(...), [Input('button', 'n_clicks')])
  def update_output(n_clicks):
      if n_clicks == 0:
          raise dash.exceptions.PreventDefault
      ...

  or, users would supply some default output properties:

  @app.callback(Output(...), [Input('button', 'n_clicks')])
  def update_output(n_clicks):
      if n_clicks == 0:
          return ''  
          # or maybe some helper text like:
          # return 'Click on the button to run the model'
      ...

• Since we would start passing default props back to the callbacks, the component's default set of properties would become part of the official component API. That is, we wouldn't be able to change these properties without it being considered a breaking change. So, if we go forward with one of these solutions, we should inspect the default properties for all of our components.

Architecture Discussions

Mechanisms to retrieve the default properties

In previous prototypes (#288), we've incorporated the serialized static prop types (serialized as part of the metadata.json files created with react-docgen) into the dynamic python classes. This solution required no changes to dash-renderer as it would include the default properties implicitly in the serialization. That is, dcc.Dropdown() would get serialized the same way as if it was specified dcc.Dropdown(multi=False, value=None, options=[]). This solution reduces the complexity in dash-renderer but it has a few flaws:

• It doesn't handle computed defaults
• It increases the size of the payload for all requests. html components have 10s of properties and this could add up for large layouts.

For similar reasons, plotly/dash-renderer#81 isn't a complete solution either. While its part of "Solution 1" above, it doesn't handle passing comptued data into the callback as State. In this solution, the components provide the computed defaults on their own schedule (by calling setProps in their component lifecycle, frequently on componentDidMount). This means that the initialization callbacks won't necessarily have these computed values, meaning that None would still be passed in as State for unspecified properties.

In order to handle computed defaults, we'll need a solution that works in dash-renderer. Here's a basic sketch of the solution:

1. (This is the current behaviour) The dev-supplied component tree is serialized in the layout redux store.

2. TreeContainer.react recursively crawls this layout object (as it does now). At each node, it would:
   a. Determine which component properties are Input or State.
   b. For those properties, retrieve the default properties of that component and merges them into the that node in the layout store. These default properties are static or are a function of the dev-supplied properties.
   c. Render the component with this new set of properties.

   Alternatively, b and c could be reversed: dash-renderer could render the component (letting the component itself work out its default properties) and then extract the properties from the component.

I have not investigated if there is a standard way to retrieve the default props from a component. In an ideal world, dash-renderer would be able to introspect defaultProps or call getDefaultProps. I suspect that this isn't possible without instantiating the component first. If so, we would need to extract the props after instantiating the component and we'd need to ensure that the component's default properties have been computed at this point. For example, if a component computes its default props in componentDidMount, this will not be called until the component is rendered (which, at this point in the app's lifecycle, the component is only instantiated, it hasn't been rendered yet).

If it's not possible to extract these properties via the React component classes/instances, then we could define a new Dash-specific component class method like computeDefaultProps(props) and call that before rendering. This would increase the component authoring complexity and so it would be preferable if we could avoid this.

[chriddyp]

cc @plotly/dash

[rmarren1]

In Solution 1, if we had a callback with 4 inputs where 1 is provided in the initial layout and 3 are not, would it fire? If so, would the values of those 3 be None or the default (computed or static) properties?

[sdementen]

If a callback has no Input (nor State), will it be fired?
A simple use case I have is to initialize a property within a Flask context (initalisation in the app.layout has not yet the Flask context).

But perhaps easier to have a custom object for such use case to initialize a value within a Flask context, something like:

dcc.Input(id='input-b', value=dash.FlaskContext(init_input),

with

def init_input():
    from Flask import request, g
    return request.headers.get("X-DEFAULT-VALUE", "no default provided")

instead of a the potential callback solution (if it gets fired at app startup time)

@app.callback(
    Output('input-b', 'value'),
)
def init_input(v):
    from Flask import request, g
    return request.headers.get("X-DEFAULT-VALUE", "no default provided")

[wave-ami]

What I ended up using was a simple work-around within the decorated callback (so simple, it's probably been done before but I thought I'd share anyway). If you mask the input variables hitting the function such that they can be checked with a simple conditional - similar to how @chriddyp proposed but you can chain them all into the first if statement if their value is None - then you get an immediate check and can provide the defaults in one spot. Simply lean towards whatever a benign default may be, for example 0 for n_clicks, as shown here:

...
@app.callback(
    Output('graph', 'figure'),
    [
        Input('submit_button', 'n_clicks'),
        Input('input-var', 'value')
    ]
)
def update_output(n_clicks_, input_value_):
    if n_clicks_ and input_value_:
        n_clicks, input_value = n_clicks_, input_value_
    else:
        n_clicks, input_value = 0, ''
        # OR here simply return {'data': [None], 'layout': None} which doesn't throw an exception
    ...

Other than that, I'd vote for a combination of Solutions 1 & 2 with the thinking that if you're able to set the callbacks to be/not-be self-triggering, it would be a similar mechanism to allow a default value setting.

[alexcjohnson]

Long obsolete, with prevent_initial_call and such 😄