Compilation error with concatenate layer

[rdhen3]

Hi there, I have just started using hls4ml a couple of days ago with the hope of applying it to a simple SSD inspired CNN.
The model involves some number >= 0 of convolutional feature extraction layers well as several parallel sets of predictor layers whose outputs needs to be concatenated together.

A simple version of the model has the following keras summary:

The current stable build seems to have issues in processing concatenate layers in that i think the shape is incorrectly inferred leading to an IndexError in hls4ml/model/hls_layers.py:

   1380         axis = self.attributes['axis']
   1381         shape = inp1.shape[:]
-> 1382         shape[axis] += inp2.shape[axis]
   1383         rank = len(shape)
   1384         if rank > 1:

IndexError: list index out of range

I tried pulling the latest master upon seeing issue #361 and it seems that the shape/indexing problem was solved by this and is get the following hls4ml summary of the model:

which seems to match up to the original model correctly.

However, now I have hit a compilation error when building/compiling the model:

I thought perhaps this related to the update going on in #402 so I tried swapping to that branch and trying again but received an identical error. Any thoughts?

[jmduarte]

@rdhen3 can you provide a script to reproduce the error? I do believe the latest error is related to issues being addressed in #402.

Also, to confirm are you using io_stream or io_parallel?

[rdhen3]

concatenate2d_issue_415.zip

I've attached a python script which reproduces all the errors mentioned on my end, dependent the hls4ml version. It gives an IndexError on version 0.5.0 and then gives

firmware/myproject.cpp: In function ‘void myproject(input_t*, result_t*, short unsigned int&, short unsigned int&)’:
firmware/myproject.cpp:94:25: error: ‘layer13_t’ was not declared in this scope
     nnet::concatenate2d<layer13_t, layer15_t, result_t, config18>(layer13_out, layer15_out, layer18_out); // predictions

on version 0.5.1 which (both master and tranpose2d_concat2d_stream)

In all honesty, I don't know which of io_stream or io_parallel I'm actually using I've more or less just followed the basic tutorial with the model I've attached, I'm a little hazy on the details of those in this context.

[rdhen3]

Are the Reshape layers supported?
In trying to work around this issue I was able to make it work without any Concatenate layers by just combining the classes and boxes layers into one (This would only work for this simple version of the model with only 1 predictor layer). However, I discovered what appears to be another issue arising from having a final Reshape layer.
The output layer gets None when you call hls_model.get_output_variables()[0].definition_cpp() which seems to suggest its not implemented. This causes the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
.../reshape_issue.py in <module>
    113 
    114 if __name__ == '__main__':
--> 115     main()
.../reshape_issue.py in main()
    106     print("Output names: ", name_out)
    107 
--> 108     hls_model.compile()
    109 
    110     hls_prediction = hls_model.predict(X_input)

.../hls4ml/model/hls_model.py in compile(self)
    524 
    525     def compile(self):
--> 526         self.write()
    527 
    528         curr_dir = os.getcwd()

.../hls4ml/hls4ml/model/hls_model.py in write(self)
    521         self.config.config['Stamp'] = make_stamp()
    522 
--> 523         self.config.writer.write_hls(self)
    524 
    525     def compile(self):

.../hls4ml/hls4ml/writer/vivado_writer.py in write_hls(self, model)
    671         print('Writing HLS project')
    672         self.write_project_dir(model)
--> 673         self.write_project_cpp(model)
    674         self.write_project_header(model)
    675         self.write_weights(model)
.../hls4ml/writer/vivado_writer.py in write_project_cpp(self, model)
    154             elif '//hls-fpga-machine-learning insert header' in line:
    155                 inputs_str = ', '.join([self.variable_definition_cpp(model, i, as_reference=True) for i in model_inputs])
--> 156                 outputs_str = ', '.join([self.variable_definition_cpp(model, o, as_reference=True) for o in model_outputs])
    157                 brams_str  = ', \n'.join([indent + self.variable_definition_cpp(model, b, as_reference=False) for b in model_brams])
    158                 insize_str = ', '.join(['unsigned short &const_size_in_{}'.format(i) for i in range(1, len(model_inputs) + 1)])

TypeError: sequence item 0: expected str instance, NoneType found

Here's a script that reproduces it for me.
reshape_issue.zip

Is this worth another issue or a known incompatibility?

[rdhen3]

I've now located an example of how to actually handle CNNs using the io_stream method you mention. I'll re-run these examples and see if any errors still persist because that's probably the whole issue!

[jmduarte]

OK, I think I understand the root cause here, which is the final layer in this Keras model is a Reshape, which means the output node of this HLSModel is an InplaceVariable, which is currently not really supported as an output node.

On a practical note, you can essentially remove that Reshape when you implement the model in hls4ml and it should be equivalent (all hls4ml outputs are basically unrolled into 1D arrays anyway).

But, we should also be able to support a final Reshape layer. I'll take a look into it, but @vloncar may have some advice on the best way to handle this.

[rdhen3]

Indeed I did notice the behaviour of unrolling the output to 1D. Is this done in the same way as, say, a keras Flatten layer, or by extension the same as numpy.reshape?
I believe I could indeed remove the Reshape layer but would still need to implement it outside the network the output at some point in order to decode the predictions correctly.

[jmduarte]

Yes, in your particular case, I would do this:

model = Model(inputs=x, outputs=classes_boxes)
...
keras_prediction = model.predict(X_input).reshape((X_input.shape[0], 16, 6))
...
hls_prediction = hls_model.predict(X_input).reshape((X_input.shape[0], 16, 6))