Skip to content

External weights vivado accelerator #646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

External weights vivado accelerator #646

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5de33c6
loosen EliminateLinearActivation match criteria--potentially unsafe
jmitrevs Aug 17, 2022
3295432
Extend the VivadoAccelerator backend with weights on the interface.
GiuseppeDiGuglielmo Aug 23, 2022
932fec8
Cover the case Interface is not defined
GiuseppeDiGuglielmo Aug 24, 2022
1c9929e
Generate standalone application for ZynqMP (tested on Ultra96v2)
GiuseppeDiGuglielmo Aug 25, 2022
fc00d4b
Add AXI-master Python driver for Ultra96v2
GiuseppeDiGuglielmo Sep 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions hls4ml/backends/vivado_accelerator/supported_boards.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,5 +38,11 @@
"python_drivers": {"axi_stream": "axi_stream_driver.py"},
"krnl_rtl_srcs": {"axi_stream": "krnl_rtl_src"},
"c_drivers": {}
},
"ultra96v2": {
"part": "xczu3eg-sbva484-1-e",
"tcl_scripts": {"axi_lite": "axi_lite_design.tcl", "axi_stream": "axi_stream_design.tcl", "axi_master": "axi_master_design.tcl"},
"python_drivers": {"axi_stream": "axi_master_driver.py"},
"c_drivers": { "axi_master": "axi_master_design.c"}
}
}
20 changes: 14 additions & 6 deletions hls4ml/backends/vivado_accelerator/vivado_accelerator_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,14 +128,22 @@ def get_clock_period(self):
def get_driver_path(self):
if self.board.startswith('alveo'):
return '../templates/vivado_accelerator/' + 'alveo/' + self.driver + '_drivers/' + \
self.get_driver_file()
self.get_driver_files()
else:
return '../templates/vivado_accelerator/' + self.board + '/' + self.driver + '_drivers/' + \
self.get_driver_file()

def get_driver_file(self):
driver_ext = '.py' if self.driver == 'python' else '.h'
return self.interface + '_driver' + driver_ext
self.get_driver_files()

#def get_driver_file(self):
# driver_ext = '.py' if self.driver == 'python' else '.h'
# return self.interface + '_driver' + driver_ext

def get_driver_files(self):
if self.driver == 'c':
driver_dir = 'sdk'
return driver_dir
elif self.driver == 'python':
driver_ext = '.py'
return self.interface + '_driver' + driver_ext

def get_krnl_rtl_src_dir(self):
return '../templates/vivado_accelerator/' + 'alveo/' + '/krnl_rtl_src'
Expand Down
28 changes: 23 additions & 5 deletions hls4ml/model/graph.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -604,14 +604,20 @@ def compile(self):
self._top_function_lib = ctypes.cdll.LoadLibrary(lib_name)

def _get_top_function(self, x):

io_type = self.config.get_config_value('IOType')
interface = self.config.get_config_value('AcceleratorConfig')['Interface'] if self.config.get_config_value('AcceleratorConfig') else None
config_weights = (io_type == 'io_stream') and (interface == 'axi_master')
Comment on lines +608 to +610
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things:

  • I don't think IOType of io_stream and Interface of axi_master should be the trigger for reprogrammable weights, I think it would be better as a new configuration parameter (of the AcceleratorConfig). Then there would probably need to be some assert somewhere to allow only combinations of those other parameters that have been implemented for (ie right now it also depends on the board)
  • As written this can't go here in ModelGraph as it's backend specific

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. This was mostly a placeholder, and I was going to ask what should be better.

As in the main thread, do we already have a matrix or list of the existing possible combinations (what goes with what)? I am not sure how to properly set up the configuration/trigger. Do you have suggestions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started putting together a document, which is far from being final. @thesps and @jmitrevs should be able to edit. Not sure if that may help or if it is better to keep it to the comments here. I bit of both I guess.

If I understand it right the additional configuration parameters (to enable programmable weights) should be passed via the function create_initial_config for the VivadoAccelerator backend.

If you agree, then I would add:

  • configurable_weights
    • bool, optional
    • weights are configurable at run time and thus they are exposed to the wrapper interface
    • defaults to False
  • weight_type
    • dict, optional
    • a dictionary that specifies the data type for set of layer weights (can be float or an ap_type); if the type is not specified for a layer, then it defaults to float
    • defaults to an empty dictionary that is all of the weights are float
    • note: VivadoAcceleratorBackend will round the number of bits used to the next power-of-2 value.

Please let me know if that makes sense.


if self._top_function_lib is None:
raise Exception('Model not compiled')
if len(self.get_input_variables()) == 1:
xlist = [x]
else:
xlist = x
n_outputs = len(self.get_output_variables())

n_weights = len(self.get_weight_variables())

for xi in xlist:
if not isinstance(xi, np.ndarray):
raise Exception('Expected numpy.ndarray, but got {}'.format(type(x)))
Expand All @@ -628,9 +634,9 @@ def _get_top_function(self, x):
else:
raise Exception('Invalid type ({}) of numpy array. Supported types are: single, float32, double, float64, float_.'.format(x0.dtype))


top_function.restype = None
top_function.argtypes = [npc.ndpointer(ctype, flags="C_CONTIGUOUS") for i in range(len(xlist) + n_outputs)]
top_function.argtypes = [npc.ndpointer(ctype, flags="C_CONTIGUOUS") \
for i in range(len(xlist) + (n_weights if config_weights else 0) + n_outputs)]

return top_function, ctype

Expand All @@ -654,10 +660,16 @@ def _compute_n_samples(self, x):
return int(n_sample)

def predict(self, x):

io_type = self.config.get_config_value('IOType')
interface = self.config.get_config_value('AcceleratorConfig')['Interface'] if self.config.get_config_value('AcceleratorConfig') else None
config_weights = (io_type == 'io_stream') and (interface == 'axi_master')
Comment on lines +664 to +666
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above


top_function, ctype = self._get_top_function(x)
n_samples = self._compute_n_samples(x)
n_inputs = len(self.get_input_variables())
n_outputs = len(self.get_output_variables())
n_weights = len(self.get_weight_variables())

curr_dir = os.getcwd()
os.chdir(self.config.get_output_dir() + '/firmware')
Expand All @@ -675,10 +687,16 @@ def predict(self, x):
inp = [np.asarray(xj[i]) for xj in x]
argtuple = inp
argtuple += predictions
if config_weights:
for j in range(n_weights):
weights = [float(w) for w in self.get_weight_variables()[j]]
argtuple += [np.asarray(weights)]
argtuple = tuple(argtuple)
top_function(*argtuple)
output.append(predictions)

if config_weights and n_samples == 1 and n_inputs:
output.append([predictions])
else:
output.append(predictions)

# Convert to list of numpy arrays (one for each output)
output = [np.asarray([output[i_sample][i_output] for i_sample in range(n_samples)]) for i_output in range(n_outputs)]
Expand Down
2 changes: 1 addition & 1 deletion hls4ml/model/optimizer/passes/nop.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ def match(self, node):
cast = False
if isinstance(node, Activation):
cast = node.get_input_variable().type.precision != node.get_output_variable().type.precision
return isinstance(node, Activation) and node.get_attr('activation') == 'linear' and not cast
return isinstance(node, Activation) and node.get_attr('activation') == 'linear' # and not cast
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

? I don't think this should be included in this PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was a quick and dirty hack to get some models to optimize better, but it really was meant only for that branch, where we had checked the correctness. I agree that it should not be in the PR.


def transform(self, model, node):
model.remove_node(node)
Expand Down
9 changes: 7 additions & 2 deletions hls4ml/templates/vivado/myproject_test.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@
#include <vector>
#include <map>
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <cstdlib>
#include <cmath>
#include <cfloat>
Comment on lines -25 to +27
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to include cfloat at a certain point, but I do not need it in the current version of the code.

When I was at it I changed to cmath and cstdlib, that I usually include that way when working in C++. We can discard this change.


#include "firmware/myproject.h"
#include "firmware/nnet_utils/nnet_helpers.h"
Expand Down Expand Up @@ -56,6 +57,10 @@ int main(int argc, char **argv)
std::string pline;
int e = 0;

//hls-fpga-machine-learning insert weights

//hls-fpga-machine-learning insert load weights

if (fin.is_open() && fpr.is_open()) {
while ( std::getline(fin,iline) && std::getline (fpr,pline) ) {
if (e % CHECKPOINT == 0) std::cout << "Processing input " << e << std::endl;
Expand Down
9 changes: 6 additions & 3 deletions hls4ml/templates/vivado_accelerator/myproject_axi.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,17 @@
//hls-fpga-machine-learning insert include

void myproject(
input_axi_t in[N_IN],
output_axi_t out[N_OUT]
){
input_axi_t in[N_IN]
, output_axi_t out[N_OUT]
//hls-fpga-machine-learning insert weights
){

//hls-fpga-machine-learning insert interface

//hls-fpga-machine-learning insert local vars

//hls-fpga-machine-learning insert enqueue weights

//hls-fpga-machine-learning insert enqueue

//hls-fpga-machine-learning insert call
Expand Down
7 changes: 4 additions & 3 deletions hls4ml/templates/vivado_accelerator/myproject_axi.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
//hls-fpga-machine-learning insert definitions

void myproject(
input_axi_t in[N_IN],
output_axi_t out[N_OUT]
);
input_axi_t in[N_IN]
, output_axi_t out[N_OUT]
//hls-fpga-machine-learning insert weights
);
#endif
Loading