QKeras bad predictions

[HenningCode]

Hello Guys,
with the fix from yesterday I tried the example "qkeras_mnist_dense" example model, but the predictions are pretty bad.

I then tried to create my own QKeras model and tried it for a short amount of time on the mnist dataset just to see if thats the also the case with my own model and the results were pretty bad again. Am I doing something wrong?

This is the code to test the example model: (for this I used the latest branch form the Github)

config = hls4ml.utils.fetch_example_model('qkeras_mnist_dense.json')
print_dict(config)
hls_model = hls4ml.converters.keras_to_hls(config)


(x_train, y_train), (x_test, y_test) = mnist.load_data()


RESHAPED = 28*28
NB_CLASSES = 10

x_train = x_train.astype("float32")
x_test = x_test.astype("float32")

x_train = x_train.reshape(x_train.shape[0],RESHAPED)
x_test = x_test.reshape(x_test.shape[0],RESHAPED)

#Tested with and without 
x_train /= 256 
x_test /= 256


y_train = to_categorical(y_train, NB_CLASSES)
y_test = to_categorical(y_test, NB_CLASSES)


hls_model.compile()
print('Ground truth\n',y_test[0:5])
print('HLS Predict:\n',hls_model.predict(x_test[0:5]))

This is the script I am using to test the QKeras Model. If I change the single QDense layer to a normal Dense layer the predictions of the HLS model are pretty close to the predictions of the keras model. Why is that the case? (For this I was using the pip installation)

def count_errors(x,y):
    error = 0
    for i in range(len(x)):
        if(not np.array_equal(x[i],y[i])):
            error += 1

    return error
        

print("============================================================================\n\n\n")

NB_EPOCH = 2
BATCH_SIZE = 64
VERBOSE = 1
NB_CLASSES = 10
OPTIMIZER = Adam(learning_rate=0.0001, decay=0.000025)
VALIDATION_SPLIT = 0.1
BUILDING = 0

(x_train, y_train), (x_test, y_test) = mnist.load_data()

RESHAPED = 784

x_test_orig = x_test

x_train = x_train.astype("float32")
x_test = x_test.astype("float32")


x_train = x_train.reshape(x_train.shape[0], RESHAPED)
x_test = x_test.reshape(x_test.shape[0], RESHAPED)

#Here only tested with, without the accuracy is bad even without QDense
x_train /= 256
x_test /= 256

print('Train shape: ', x_train.shape)
print('Test shape: ', x_test.shape)

y_train = to_categorical(y_train, NB_CLASSES)
y_test = to_categorical(y_test, NB_CLASSES)


x = x_in = Input((RESHAPED,), name="input")
#x = Dense(64,name="dense0")(x)
x = QDense(64,kernel_quantizer=quantized_bits(16,6),
        bias_quantizer=quantized_bits(16,6),name="dense0")(x)
x = Activation("relu", name="act0")(x)
x = Dense(NB_CLASSES,name="dense2")(x)
x = Activation("softmax", name="softmax")(x)

model = Model(inputs=[x_in], outputs=[x])
model.summary()
model.compile(
    loss="categorical_crossentropy", optimizer=OPTIMIZER, metrics=["accuracy"])

history = model.fit(
    x_train, y_train, batch_size=BATCH_SIZE,
    epochs=NB_EPOCH, initial_epoch=1, verbose=VERBOSE,
    validation_split=VALIDATION_SPLIT)

config = hls4ml.utils.config_from_keras_model(model,granularity='name')
config['Model']['Strategy'] = 'Resource'
print_dict(config)
hls_model = hls4ml.converters.convert_from_keras_model(model,
                                            hls_config=config,
                                            output_dir='../output/model_std/hls4ml_prj',
                                            fpga_part='xc7z020clg400-1')
_ = hls_model.compile()

TEST_CASES = 5

out_model = model.predict(x_test[0:TEST_CASES])
out_model_change = np.zeros_like(out_model)
out_model_change[np.arange(len(out_model)), out_model.argmax(1)] = 1

print("Output of Normal Model:\n", out_model)

out_hls = hls_model.predict(x_test[0:TEST_CASES])
out_hls_change = np.zeros_like(out_hls)
out_hls_change[np.arange(len(out_hls)), out_hls.argmax(1)] = 1

print("Output of HLS Model:\n", out_hls)

print('Error Normal: ', count_errors(out_model_change,y_test[0:TEST_CASES]))
print('Error HLS: ', count_errors(out_hls_change,y_test[0:TEST_CASES]))

[ChiRuiChen]

Hi, I had tested your code. It seems fine to me since the numbers are small.
What are your outputs?

[HenningCode]

Okay thats kinda weird.. These are my Results for the QDense Layer.

And these are Results if I comment out the QDense Layer and use the Dense layer.

[ChiRuiChen]

Actually, the quantized_bit you set has an extra bit for integer.
QKeras doesn't count the sign-bit when we specify the number of bits, so your Qdense uses <16,7> instead of <16,6>.
maybe it's the cause?
try set the quantized_bit = (16,5,alpha=1)

[HenningCode]

Okay that seems to work, but it was more related to the alpha=1. The Bit size isn't a factor.
Did your results come from the dense only model or why did you get good results?

But thanks already!!

[ChiRuiChen]

my results come from Qdense, all are the same as your source code.
So did set the quantized_bit = (16,5,alpha=1) help?

[HenningCode]

Yes it did help! I was just wondering why it did work for you and not for me.

[thesps]

It seems like this might be solved?

I'd add that alpha = 1 often helps. alpha != 1 is allowed, but you'd expect to see an ApplyAlpha layer inserted after the Dense layer in the HLSModel. You should see it if you plot the model. It's needed to restore the weight scale factors to the layer output, but the precision it uses may need to be set manually.

When seeing disagreement between QKeras (and anything) and hls4ml, the first thing is usually to profile and trace the model to find which layer is the first to have a disagreement. In your case, I'd suspect it's the ApplyAlphas.

[HenningCode]

Not quite solved. Now I tried to use QActivations and if I use them, the results are as bad as shown above. Do I have to add something there too?

x = QActivation("quantized_relu(4,0)", name="act0_m")(x)

But I will try to dig into it with trace and profiling!

[jmduarte]

What branches/versions of hls4ml and QKeras are you using?

[HenningCode]

I used the pip installation of hls4ml and also tried it with the master branch. So it was Version 0.5.0 and 0.5.1.
Qkeras was always installed like this git+https://github.com/google/qkeras.git#egg=qkeras

[thesps]

The issue with quantized_relu could be related to #441. Could you try out either the release-v0.6.0 branch or master + #441?

[HenningCode]

I tried it with release 0.6.0 and did not improve the outcome.

[HenningCode]

Now I tried it with PR #441 and I got no improvement.
I used this command to install the PR pip install git+https://github.com/fastmachinelearning/hls4ml.git@refs/pull/441/merge

and this script to try this out:

import hls4ml
import tensorflow.keras.backend as K
from tensorflow.keras.datasets import mnist
from tensorflow.keras.layers import Activation
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import *
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import to_categorical

import yaml

import matplotlib.pyplot as plt

from qkeras import *

def print_dict(d, indent=0):
    align = 20
    for key, value in d.items():
        print('  ' * indent + str(key), end='')
        if isinstance(value, dict):
            print()
            print_dict(value, indent+1)
        else:
            print(':' + ' ' * (20 - len(key) - 2 * indent) + str(value))


def count_errors(x, y):
    error = 0
    for i in range(len(x)):
        if(not np.array_equal(x[i], y[i])):
            error += 1

    return error


def yaml_save(config, filepath):
    with open(filepath, 'w') as outfile:
        yaml.dump(config, outfile, default_flow_style=False)


print("============================================================================\n\n\n")


model_dir = 'model/qkaras_pull.h5'
config_dir ='config/qkaras_pull_config.yml'
picture_dir = 'picture/qkaras_pull.png'

NB_EPOCH = 10
BATCH_SIZE = 64
VERBOSE = 1
NB_CLASSES = 10
OPTIMIZER = Adam(learning_rate=0.0001, decay=0.000025)
VALIDATION_SPLIT = 0.1

SAFE_CONFIG = 1
COVNET = 1

(x_train, y_train), (x_test, y_test) = mnist.load_data()

RESHAPED = 784

x_test_orig = x_test

x_train = x_train.astype("float32")
x_test = x_test.astype("float32")

if COVNET:
    x_train = x_train[..., np.newaxis]
    x_test = x_test[..., np.newaxis]
else:
    x_train = x_train.reshape(x_train.shape[0], RESHAPED)
    x_test = x_test.reshape(x_test.shape[0], RESHAPED)

# Here only tested with, without the accuracy is bad even without QDense
x_train /= 256
x_test /= 256

print('Train shape: ', x_train.shape)
print('Test shape: ', x_test.shape)

y_train = to_categorical(y_train, NB_CLASSES)
y_test = to_categorical(y_test, NB_CLASSES)

if COVNET:
    x = x_in = Input(
        x_train.shape[1:-1] + (1,), name="input")
    x = QConv2D(
        16, (2, 2), strides=(2, 2),
        kernel_quantizer=quantized_bits(4, 0, alpha=1),
        bias_quantizer=quantized_bits(4, 0, alpha=1),
        name="conv2d_0_m")(x)
    x = QActivation("quantized_relu(4,0)", name="act0_m")(x)
    x = QConv2D(
        32, (3, 3), strides=(2, 2),
        kernel_quantizer=quantized_bits(4, 0, alpha=1),
        bias_quantizer=quantized_bits(4, 0, alpha=1),
        name="conv2d_1_m")(x)
    x = QActivation("quantized_relu(4,0)", name="act1_m")(x)
    x = Flatten()(x)
    x = QDense(NB_CLASSES, kernel_quantizer=quantized_bits(4, 0, alpha=1),
               bias_quantizer=quantized_bits(4, 0, 1, alpha=1),
               name="dense")(x)
    x = Activation("softmax", name="softmax")(x)


else:
    x = x_in = Input((RESHAPED,), name="input")
    #x = Dense(64,name="dense0")(x)
    x = QDense(64, kernel_quantizer=quantized_bits(5, 0, alpha=1),
               bias_quantizer=quantized_bits(5, 0, alpha=1), name="dense0")(x)
    x = QBatchNormalization()(x)
    x = QActivation("quantized_relu(4,2)", name="act0")(x)
    x = QDense(NB_CLASSES, kernel_quantizer=quantized_bits(5, 0, alpha=1),
               bias_quantizer=quantized_bits(5, 0, alpha=1), name="dense2")(x)
    x = Activation("softmax", name="softmax")(x)

model = Model(inputs=[x_in], outputs=[x])
model.summary()
model.compile(
    loss="categorical_crossentropy", optimizer=OPTIMIZER, metrics=["accuracy"])

history = model.fit(
    x_train, y_train, batch_size=BATCH_SIZE,
    epochs=NB_EPOCH, initial_epoch=1, verbose=VERBOSE,
    validation_split=VALIDATION_SPLIT)

model.save(model_dir)

config = hls4ml.utils.config_from_keras_model(model, granularity='name')
config['Model']['Strategy'] = 'Resource'

if SAFE_CONFIG:
    yaml_save(config=config, filepath=config_dir)


print_dict(config)
hls_model = hls4ml.converters.convert_from_keras_model(model,
                                                       hls_config=config,
                                                       output_dir='../output/model_std/hls4ml_prj')
_ = hls_model.compile()


TEST_CASES = 1000

hls4ml.utils.plot_model(hls_model, show_shapes=True,
                        show_precision=True, to_file='pictures/batchdense_model.png')
plt.show()

out_model = model.predict(x_test[0:TEST_CASES])
out_model_change = np.zeros_like(out_model)
out_model_change[np.arange(len(out_model)), out_model.argmax(1)] = 1

out_hls = hls_model.predict(x_test[0:TEST_CASES])
out_hls_change = np.zeros_like(out_hls)
out_hls_change[np.arange(len(out_hls)), out_hls.argmax(1)] = 1

print('Error Normal: ', count_errors(out_model_change, y_test[0:TEST_CASES]))
print('Error HLS: ', count_errors(out_hls_change, y_test[0:TEST_CASES]))