Activation and Quantization Functions Tutorial
This notebook demonstrates the usage of various quantization and activation functions provided by QKerasV3 (a quantization extension for Keras). QKerasV3 is widely used for creating ultra-low-power, quantized neural networks suitable for edge hardware or FPGA deployment.
Prerequisites
Make sure you have qkeras-v3 and numpy installed. If you are running this in Colab, uncomment the line below to install it:
# !pip install qkeras-v3 numpy tensorflow
import numpy as np
from qkeras import (
bernoulli,
binary,
hard_sigmoid,
hard_tanh,
quantized_bits,
quantized_po2,
quantized_relu,
quantized_relu_po2,
quantized_tanh,
set_internal_sigmoid,
smooth_sigmoid,
smooth_tanh,
stochastic_binary,
stochastic_ternary,
ternary,
)
/Users/mariuskoppel/cms/qkeras/venv/lib/python3.11/site-packages/keras/src/export/tf2onnx_lib.py:8: FutureWarning: In the future `np.object` will be defined as the corresponding NumPy scalar.
if not hasattr(np, "object"):
Imports successful!
1. Stochastic Rounding with Power-of-2 (PO2) Quantizers
Stochastic rounding approximates continuous numbers on average by rounding up or down probabilistically based on the distance to the nearest target values. Here, we verify that the mean value of a large sample size matches the original value using quantized_po2 and quantized_relu_po2.
count = 100000
val = 42
a = np.array([val] * count)
# 1. Test Power of 2 (PO2) Stochastic Rounding
b = quantized_po2(use_stochastic_rounding=True)(a)
res = np.sum(b.numpy()) / count
print(f"PO2 Stochastic Mean: {res} (should be close to {val})")
# 2. Test ReLU PO2 Stochastic Rounding
b = quantized_relu_po2(use_stochastic_rounding=True)(a)
res = np.sum(b.numpy()) / count
print(f"ReLU PO2 Stochastic Mean: {res} (should be close to {val})")
# 3. Test Negative bounds for ReLU PO2 (Should zero out negative inputs)
a_neg = np.array([-1] * count)
b_neg = quantized_relu_po2(use_stochastic_rounding=True)(a_neg)
res_neg = np.sum(b_neg.numpy()) / count
print(f"ReLU PO2 Negative Bound Mean: {res_neg} (should be exactly 0)")
PO2 Stochastic Mean: 32.0 (should be close to 42)
ReLU PO2 Stochastic Mean: 32.0 (should be close to 42)
ReLU PO2 Negative Bound Mean: 0.0 (should be exactly 0)
2. Standard Non-Stochastic Quantization & Approximations
Let’s observe how baseline fixed-point quantizers (quantized_bits, quantized_relu) and smooth/hard activation function variations affect a small array of numbers.
# Testing standard activation functions
a = np.array([0.194336])
print(f"Original 'a': {a.astype('float16')}")
print(f"quantized_relu: {quantized_relu(6, 2)(a).numpy().astype('float16')}")
print(f"smooth_sigmoid: {smooth_sigmoid(a).numpy().astype('float16')}")
print(f"hard_sigmoid: {hard_sigmoid(a).numpy().astype('float16')}")
print(f"hard_tanh: {hard_tanh(a).numpy().astype('float16')}")
print(f"smooth_tanh: {smooth_tanh(a).numpy().astype('float16')}")
print("\n--- Testing quantized_bits configurations ---")
# Testing variations of quantized_bits(bits, integer, keep_negative)
c = np.array(np.arange(-1.5, 1.51, 0.3), dtype="float32")
print(f"Original 'c': {c.astype('float16')}")
print(f"qb_111: {quantized_bits(1, 1, 1)(c).numpy().astype('float16')}")
print(f"qb_210: {quantized_bits(2, 1, 0)(c).numpy().astype('float16')}")
print(f"qb_211: {quantized_bits(2, 1, 1)(c).numpy().astype('float16')}")
print(f"qb_300: {quantized_bits(3, 0, 0)(c).numpy().astype('float16')}")
print(f"qb_301: {quantized_bits(3, 0, 1)(c).numpy().astype('float16')}")
Original 'a': [0.1943]
quantized_relu: [0.1875]
smooth_sigmoid: [0.5366]
hard_sigmoid: [0.597]
hard_tanh: [0.1943]
smooth_tanh: [0.0729]
--- Testing quantized_bits configurations ---
Original 'c': [-1.5 -1.2 -0.9 -0.6 -0.3 0. 0.3 0.6 0.9 1.2 1.5]
qb_111: [-1. -1. -1. -1. -1. 1. 1. 1. 1. 1. 1.]
qb_210: [-2. -1. -1. -1. 0. 0. 0. 1. 1. 1. 1.]
qb_211: [-1. -1. -1. -1. 0. 0. 0. 1. 1. 1. 1.]
qb_300: [-1. -1. -1. -0.5 -0.25 0. 0.25 0.5 0.75 0.75 0.75]
qb_301: [-0.75 -0.75 -0.75 -0.5 -0.25 0. 0.25 0.5 0.75 0.75 0.75]
3. Advanced Stochastic Activations (Bernoulli & Ternary)
This section demonstrates how stochastic behavior acts over 1,000 duplicated evaluations of our target range c. On average, the stochastic outputs approximate deterministic profiles like hard_sigmoid.
c_1000 = np.array(np.array([list(c)] * 1000), dtype="float32")
print("--- Bernoulli Quantization ---")
b_bernoulli = np.sum(bernoulli()(c_1000).numpy().astype(np.int32), axis=0) / 1000.0
print(f"hard_sigmoid baseline: {hard_sigmoid(c).numpy().astype('float16')}")
print(f"Bernoulli (Mean): {b_bernoulli.astype('float16')}")
print("\n--- Stochastic Ternary Quantization ---")
t_ternary = stochastic_ternary(alpha="auto")(c_1000).numpy()
for i in range(3): # Printing first 3 samples for brevity
print(f"stochastic_ternary sample ({i}): {t_ternary[i]}")
st_all = np.round(np.sum(t_ternary.astype("float32"), axis=0).astype("float16") / 1000.0, 2).astype("float16")
print(f"Stochastic Ternary (Mean): {st_all}")
--- Bernoulli Quantization ---
hard_sigmoid baseline: [0. 0. 0.05 0.2 0.35 0.5 0.65 0.8 0.95 1. 1. ]
Bernoulli (Mean): [0. 0.002 0.004 0.029 0. 0.488 0.858 0.969 0.996 1. 0.999]
--- Stochastic Ternary Quantization ---
stochastic_ternary sample (0): [-1.4999999e+00 -1.1999999e+00 -8.9999789e-01 -5.9999996e-01
-2.9999998e-01 2.2204458e-16 2.9999998e-01 5.9999996e-01
9.0000802e-01 1.2000002e+00 1.4999999e+00]
stochastic_ternary sample (1): [-1.4999999e+00 -1.1999999e+00 -8.9999789e-01 -5.9999996e-01
-2.9999998e-01 2.2204458e-16 2.9999998e-01 5.9999996e-01
9.0000802e-01 1.2000002e+00 1.4999999e+00]
stochastic_ternary sample (2): [-1.4999999e+00 -1.1999999e+00 -8.9999789e-01 -5.9999996e-01
-2.9999998e-01 2.2204458e-16 2.9999998e-01 5.9999996e-01
9.0000802e-01 1.2000002e+00 1.4999999e+00]
Stochastic Ternary (Mean): [-1.5 -1.2 -0.9 -0.6 -0.3 0. 0.3 0.6 0.9 1.2 1.5]
4. Comparing Internal Sigmoid Flavors
QKerasV3 allows you to change global internal configurations. Let’s compare quantized_relu and quantized_tanh under three different internal sigmoid states: smooth, real, and hard.
flavors = ["smooth", "real", "hard"]
for flavor in flavors:
set_internal_sigmoid(flavor)
print(f"\n================= Internal Sigmoid: {flavor.upper()} =================")
print(f"qr_101: {quantized_relu(1, 0, 1)(c).numpy().astype('float16')}")
print(f"qr_111: {quantized_relu(1, 1, 1)(c).numpy().astype('float16')}")
print(f"qr_201: {quantized_relu(2, 0, 1)(c).numpy().astype('float16')}")
print(f"qr_211: {quantized_relu(2, 1, 1)(c).numpy().astype('float16')}")
print(f"qt_200: {quantized_tanh(2, 0)(c).numpy().astype('float16')}")
print(f"qt_210: {quantized_tanh(2, 1)(c).numpy().astype('float16')}")
print(f"qt_201: {quantized_tanh(2, 0, 1)(c).numpy().astype('float16')}")
print(f"qt_211: {quantized_tanh(2, 1, 1)(c).numpy().astype('float16')}")
================= Internal Sigmoid: SMOOTH =================
qr_101: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.5]
qr_111: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
qr_201: [0. 0. 0. 0. 0. 0. 0. 0. 0.5 0.5 0.5]
qr_211: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
qt_200: [-0.5 -0.5 -0.5 0. 0. 0. 0. 0. 0.5 0.5 0.5]
qt_210: [-0.5 -0.5 -0.5 0. 0. 0. 0. 0. 0.5 0.5 0.5]
qt_201: [-0.5 -0.5 -0.5 0. 0. 0. 0. 0. 0.5 0.5 0.5]
qt_211: [-0.5 -0.5 -0.5 0. 0. 0. 0. 0. 0.5 0.5 0.5]
================= Internal Sigmoid: REAL =================
qr_101: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.5 0.5]
qr_111: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
qr_201: [0. 0. 0. 0. 0. 0. 0. 0.5 0.5 0.5 0.5]
qr_211: [0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1.]
qt_200: [-0.5 -0.5 -0.5 -0.5 0. 0. 0. 0.5 0.5 0.5 0.5]
qt_210: [-0.5 -0.5 -0.5 -0.5 0. 0. 0. 0.5 0.5 0.5 0.5]
qt_201: [-0.5 -0.5 -0.5 -0.5 0. 0. 0. 0.5 0.5 0.5 0.5]
qt_211: [-0.5 -0.5 -0.5 -0.5 0. 0. 0. 0.5 0.5 0.5 0.5]
================= Internal Sigmoid: HARD =================
qr_101: [0. 0. 0. 0. 0. 0. 0. 0.5 0.5 0.5 0.5]
qr_111: [0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1.]
qr_201: [0. 0. 0. 0. 0. 0. 0.5 0.5 0.75 0.75 0.75]
qr_211: [0. 0. 0. 0. 0. 0. 0. 1. 1. 1. 1.5]
qt_200: [-1. -1. -1. -0.5 -0.5 0. 0.5 0.5 0.5 0.5 0.5]
qt_210: [-1. -1. -1. -0.5 -0.5 0. 0.5 0.5 0.5 0.5 0.5]
qt_201: [-0.5 -0.5 -0.5 -0.5 -0.5 0. 0.5 0.5 0.5 0.5 0.5]
qt_211: [-0.5 -0.5 -0.5 -0.5 -0.5 0. 0.5 0.5 0.5 0.5 0.5]
5. Power-of-2 and Binary Rounding Dynamics
Finally, let’s explore deep power-of-2 distributions alongside binary stochastic variations. This shows how aggressively values break down to single-bit or binary-state boundaries.
# Resetting back to default hard setting for final tests
set_internal_sigmoid("hard")
print("--- Deep Power-of-2 (PO2) Outputs ---")
print(f"Original 'c': {c.astype('float16')}")
print(f"q2_31: {quantized_po2(3, 1)(c).numpy().astype('float16')}")
print(f"q2_32: {quantized_po2(3, 2)(c).numpy().astype('float16')}")
print(f"qr2_21: {quantized_relu_po2(2, 1)(c).numpy().astype('float16')}")
print(f"qr2_22: {quantized_relu_po2(2, 2)(c).numpy().astype('float16')}")
print(f"qr2_32_2: {quantized_relu_po2(32, 2)(c).numpy().astype('float16')}")
print("\n--- Stochastic Binary Evaluation ---")
b_binary = stochastic_binary()(c_1000).numpy().astype(np.int32)
sbinary_mean = np.round(np.sum(b_binary, axis=0) / 1000.0, 2).astype("float16")
print(f"Stochastic Binary (Mean): {sbinary_mean}")
print(f"Deterministic Binary: {binary()(c).numpy().astype(np.int32)}")
print("\n--- Distribution Over Iterations (Stochastic Rounding = 1) ---")
print("Evaluating variations over consecutive calls:")
for i in range(3):
print(f" Run {i} - po2: {quantized_po2(use_stochastic_rounding=1)(c).numpy().astype(np.int32)}")
print(f" Run {i} - relu_po2: {quantized_relu_po2(use_stochastic_rounding=1)(c).numpy().astype(np.int32)}")
--- Deep Power-of-2 (PO2) Outputs ---
Original 'c': [-1.5 -1.2 -0.9 -0.6 -0.3 0. 0.3 0.6 0.9 1.2 1.5]
q2_31: [-1. -1. -1. -0.5 -0.25 0.0625 0.25 0.5 1.
1. 1. ]
q2_32: [-2. -1. -1. -0.5 -0.25 0.25 0.25 0.5 1. 1. 2. ]
qr2_21: [0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.25 0.5 1. 1.
1. ]
qr2_22: [0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.5 1. 1. 2. ]
qr2_32_2: [0. 0. 0. 0. 0. 0. 0.25 0.5 1. 1. 2. ]
--- Stochastic Binary Evaluation ---
Stochastic Binary (Mean): [-1. -1. -1. -1. -1. 1. 1. 1. 1. 1. 1.]
Deterministic Binary: [-1 -1 -1 -1 -1 1 1 1 1 1 1]
--- Distribution Over Iterations (Stochastic Rounding = 1) ---
Evaluating variations over consecutive calls:
Run 0 - po2: [-2 -1 -1 0 0 0 0 0 1 1 2]
Run 0 - relu_po2: [0 0 0 0 0 0 0 0 1 1 2]
Run 1 - po2: [-2 -1 -1 0 0 0 0 0 1 1 2]
Run 1 - relu_po2: [0 0 0 0 0 0 0 0 1 1 2]
Run 2 - po2: [-2 -1 -1 0 0 0 0 0 1 1 2]
Run 2 - relu_po2: [0 0 0 0 0 0 0 0 1 1 2]