Accuracy drop in quantized keras model after hls compile

[wilfredkisku]

I am working on a skip connection-based CNN structure that either uses an Add or a Concatenate function to merge different paths in a model. I am able to obtain proper accuracy in the baseline model (fp32) for both keras and hls compiled model.

But there is an issue that pops up when I use the quantized model and then convert it to hls (issue in the accuracy drop of the quantilized hls model after it is compiled).

Model Architecture:

from qkeras import QActivation
from qkeras import QDense, QConv2DBatchnorm

neurons_per_dense_layer = [16,32,64]

IFM = 6
WGT = 14

def QResNet9(input_shape = (32,32,1), classes = 10):
    img_input = Input(shape=input_shape)

    #-------------------------------------------------------------------------------
    x = QActivation('quantized_relu('+str(IFM)+',0)',name='relu_in')(img_input)
    #-------------------------------------------------------------------------------
    x = QConv2DBatchnorm(16, kernel_size = (3,3), strides=(1,1),
                         kernel_quantizer="quantized_bits("+str(WGT)+",0,alpha=1)",
                         kernel_initializer='lecun_uniform',  
                         kernel_regularizer=l1(0.0001), name='conv1', use_bias = True, padding='same')(x)
    x = QActivation('quantized_relu('+str(IFM)+',0)', name='relu_conv1')(x)

    x = QConv2DBatchnorm(16, kernel_size = (3,3), strides=(1,1),
                         kernel_quantizer="quantized_bits("+str(WGT)+",0,alpha=1)",
                         kernel_initializer='lecun_uniform',  
                         kernel_regularizer=l1(0.0001), name='conv2', use_bias = True, padding='same')(x)
    x = QActivation('quantized_relu('+str(IFM)+',0)', name='relu_conv2')(x)
    x = MaxPooling2D(pool_size=(2, 2), name='pool1')(x)
    #-------------------------------------------------------------------------------
    x_skip = x
    #-------------------------------------------------------------------------------
    x = QConv2DBatchnorm(16, kernel_size = (3,3), strides=(1,1),
                         kernel_quantizer="quantized_bits("+str(WGT)+",0,alpha=1)",
                         kernel_initializer='lecun_uniform',  
                         kernel_regularizer=l1(0.0001), name='conv3', use_bias = True, padding='same')(x)
    x = QActivation('quantized_relu('+str(IFM)+',0)', name='relu_conv3')(x)

    x = QConv2DBatchnorm(16, kernel_size = (3,3), strides=(1,1),
                         kernel_quantizer="quantized_bits("+str(WGT)+",0,alpha=1)",
                         kernel_initializer='lecun_uniform',  
                         kernel_regularizer=l1(0.0001), name='conv4', use_bias = True, padding='same')(x)
    x = QActivation('quantized_relu('+str(IFM)+',0)', name='relu_conv4')(x)
    #-------------------------------------------------------------------------------
    x = Concatenate()([x, x_skip])
    #-------------------------------------------------------------------------------
    x = QConv2DBatchnorm(16, kernel_size = (3,3), strides=(1,1),
                         kernel_quantizer="quantized_bits("+str(WGT)+",0,alpha=1)",
                         kernel_initializer='lecun_uniform',  
                         kernel_regularizer=l1(0.0001), name='conv5', use_bias = True, padding='same')(x)
    x = QActivation('quantized_relu('+str(IFM)+',0)', name='relu_conv5')(x)
    x = MaxPooling2D(pool_size=(2, 2), name='pool2')(x)

    x = QConv2DBatchnorm(16, kernel_size = (3,3), strides=(1,1),
                         kernel_quantizer="quantized_bits("+str(WGT)+",0,alpha=1)",
                         kernel_initializer='lecun_uniform',  
                         kernel_regularizer=l1(0.0001), name='conv6', use_bias = True, padding='same')(x)
    x = QActivation('quantized_relu('+str(IFM)+',0)', name='relu_conv6')(x)
    x = MaxPooling2D(pool_size=(2, 2), name='pool3')(x)
    #-------------------------------------------------------------------------------
    x_skip = x
    #-------------------------------------------------------------------------------
    x = QConv2DBatchnorm(16, kernel_size = (3,3), strides=(1,1),
                         kernel_quantizer="quantized_bits("+str(WGT)+",0,alpha=1)",
                         kernel_initializer='lecun_uniform',  
                         kernel_regularizer=l1(0.0001), name='conv7', use_bias = False, padding='same')(x)
    x = QActivation('quantized_relu('+str(IFM)+',0)', name='relu_conv7')(x)
  
    x = QConv2DBatchnorm(16, kernel_size = (3,3), strides=(1,1),
                         kernel_quantizer="quantized_bits("+str(WGT)+",0,alpha=1)",
                         kernel_initializer='lecun_uniform',  
                         kernel_regularizer=l1(0.0001), name='conv8', use_bias = False, padding='same')(x)
    x = QActivation('quantized_relu('+str(IFM)+',0)', name='relu_conv8')(x)
    #-------------------------------------------------------------------------------
    x = Concatenate()([x, x_skip])
    #-------------------------------------------------------------------------------
    x = MaxPooling2D(name='pool4')(x)
    #-------------------------------------------------------------------------------
    x = Flatten()(x)
    #-------------------------------------------------------------------------------
    for i,n in enumerate(neurons_per_dense_layer):
        print( ('Adding QDense block {} with N={} neurons').format(i,n) )
        x = QDense(n,
            kernel_quantizer="quantized_bits("+str(WGT)+",0,alpha=1)",
            kernel_initializer='lecun_uniform', kernel_regularizer=l1(0.0001),name='dense_%i_2'%i, use_bias=False)(x)
        x = BatchNormalization(name='bn_dense_{}_2'.format(i))(x)
        x = QActivation('quantized_relu('+str(IFM)+',0)',name='dense_act_%i_2'%i)(x)

    x = Dense(int(n_classes),name='output_dense')(x)
    x_out = Activation('softmax',name='output_softmax')(x)
    #----------------------------------------------------------------------
    qmodel = Model(inputs=[img_input], outputs=[x_out], name='qkeras')
    return qmodel

qmodel9 = QResNet9()
qmodel9.summary()

HLS configuration code:

# Then the QKeras model
hls4ml.model.optimizer.OutputRoundingSaturationMode.layers = ['Activation']
hls4ml.model.optimizer.OutputRoundingSaturationMode.rounding_mode = 'AP_RND'
hls4ml.model.optimizer.OutputRoundingSaturationMode.saturation_mode = 'AP_SAT'

hls_config_q = hls4ml.utils.config_from_keras_model(qmodel, granularity='name')
hls_config_q['Model']['Strategy'] = 'Resource'
hls_config_q['Model']['ReuseFactor'] = 1
hls_config_q['Model']['Precision'] = 'ap_fixed<16,6>'

hls_config_q['LayerName']['conv1']['Strategy'] = 'Resource'
hls_config_q['LayerName']['conv1']['ReuseFactor'] = 3

hls_config_q['LayerName']['conv2']['Strategy'] = 'Resource'
hls_config_q['LayerName']['conv2']['ReuseFactor'] = 2

hls_config_q['LayerName']['conv3']['Strategy'] = 'Resource'
hls_config_q['LayerName']['conv3']['ReuseFactor'] = 2

hls_config_q['LayerName']['conv4']['Strategy'] = 'Resource'
hls_config_q['LayerName']['conv4']['ReuseFactor'] = 2

hls_config_q['LayerName']['conv5']['Strategy'] = 'Resource'
hls_config_q['LayerName']['conv5']['ReuseFactor'] = 2

hls_config_q['LayerName']['conv7']['Strategy'] = 'Resource'
hls_config_q['LayerName']['conv7']['ReuseFactor'] = 2

hls_config_q['LayerName']['conv6']['Strategy'] = 'Resource'
hls_config_q['LayerName']['conv6']['ReuseFactor'] = 2

hls_config_q['LayerName']['conv8']['Strategy'] = 'Resource'
hls_config_q['LayerName']['conv8']['ReuseFactor'] = 2

hls_config_q['LayerName']['dense_0_2']['Strategy'] = 'Resource'
hls_config_q['LayerName']['dense_0_2']['ReuseFactor'] = 2

hls_config_q['LayerName']['dense_1_2']['Strategy'] = 'Resource'
hls_config_q['LayerName']['dense_1_2']['ReuseFactor'] = 2

hls_config_q['LayerName']['dense_2_2']['Strategy'] = 'Resource'
hls_config_q['LayerName']['dense_2_2']['ReuseFactor'] = 2

hls_config_q['LayerName']['output_dense']['Strategy'] = 'Resource'
hls_config_q['LayerName']['output_dense']['ReuseFactor'] = 2

hls_config_q['LayerName']['output_softmax']['Strategy'] = 'Stable'
plotting.print_dict(hls_config_q)
  
cfg_q = hls4ml.converters.create_config(backend='Vivado')
cfg_q['IOType']     = 'io_stream' # Must set this if using CNNs!
cfg_q['HLSConfig']  = hls_config_q
cfg_q['KerasModel'] = qmodel
cfg_q['OutputDir']  = 'quantized_cnn_model_C_9/'
cfg_q['XilinxPart'] = 'xczu7ev-ffvc1156-2-e'
#cfg_q['XilinxPart'] = 'xcu250-figd2104-2L-e'
  
hls_model_q = hls4ml.converters.keras_to_hls(cfg_q)
hls_model_q.compile()

Accuracy Report:

Accuracy Keras (baseline)  :  0.724
Accuracy hls4ml (baseline): 0.6976666666666667
Accuracy Keras (Qkeras)    :  0.6863333333333334
Accuracy hls4ml (Qkeras)  : 0.09866666666666667

You can see that there is a considerable drop in accuracy after HLS conversion. Can anyone point out the mistake on my part?

[jmduarte]

Could you try profiling each layer and comparing the expected output (from the QKeras model) to the actual output (from the hls4ml QKeras model)?

My suspicion is the skip connection/merge layers/max pooling may be causing overflows, especially if they're using different input/output data types.

An example for how to do this is here: https://github.com/hls4ml-finn-mlperftiny/CIFAR10/blob/main/hls4ml/convert.py#L222-L233

[wilfredkisku]

Thanks for the suggestion, I tried to profile the layers but am getting an error Graph Disconnected.

ValueError: Graph disconnected: cannot obtain value for tensor KerasTensor(type_spec=TensorSpec(shape=(None, 32, 32, 1), dtype=tf.float32, name='input_1'), name='input_1', description="created by layer 'input_1'") at layer "prune_low_magnitude_conv1". The following previous layers were accessed without issue: []

[liuhao-97]

Hi have you solve your problem?

[wilfredkisku]

@liuhao-97 the problem seems to be there only in the quantized model. The keras model that has configured with hls4ml for a precision of hls_config['Model']['Precision'] = 'ap_fixed<16,6>' does not show any accuracy drop. So, as mentioned my @jmduarte a specific operation or layer might be causing an overflow that might be dropping the accuracy of the quantized models.