CIFAR-10 Network with Ultra-Low Precision PO2 Quantization

This notebook demonstrates how to design a Quantization-Aware Training (QAT) Convolutional Neural Network using QKeras targeting the CIFAR-10 image dataset.

Core Architectural Concepts Shown in This Example

• CIFAR-10 Spatial Scaling: Unlike the grayscale $28 \times 28 \times 1$ MNIST dataset, CIFAR-10 contains complex $32 \times 32 \times 3$ RGB color images. The network uses aggressive 128 and 256 filter channels to combat structural complexity under low precision constraints.

• Ternary Non-Linearity Scaling (ternary()): Layer act0_m uses a ternary activation function. This limits the forward-passing feature maps down to just three explicit values: ${-1, 0, 1}$. This is incredibly useful for custom hardware designs (like FPGAs), converting continuous floating-point maps into standard logical control bits.

• Quantized $\mu$-law Nonlinearity (quantized_ulaw): The output classifier uses logarithmic $\mu$-law quantization. This non-uniform quantization method allocates higher bit-resolution around values closer to zero, allowing fine-grained decision clustering during final logic processing.

1. Environment Setup & Dependency Installation

Uncomment the cell below if you need to install the required dependencies in your notebook runtime environment.

# !pip install qkeras-v3 keras tensorflow

2. Imports and Initial Configuration Setup

import os
import numpy as np
import keras.ops.numpy as knp
from keras import ops
from keras.datasets import cifar10
from keras.layers import *
from keras.models import Model
from keras.optimizers import *
from keras.utils import to_categorical

from qkeras import *

# Global training hyperparameters
NB_EPOCH = 50
BATCH_SIZE = 64
VERBOSE = 1
NB_CLASSES = 10
OPTIMIZER = Adam(learning_rate=0.0001)
VALIDATION_SPLIT = 0.1

3. Load and Normalize CIFAR-10 Dataset

(x_train, y_train), (x_test, y_test) = cifar10.load_data()

x_train = x_train.astype(float)
x_test = x_test.astype(float)

x_train /= 255.0
x_test /= 255.0

print(x_train.shape[0], "train samples")
print(x_test.shape[0], "test samples")
print("Sample training labels shape:", y_train.shape)

y_train = to_categorical(y_train, NB_CLASSES)
y_test = to_categorical(y_test, NB_CLASSES)

/Users/mariuskoppel/cms/qkeras/venv/lib/python3.11/site-packages/keras/src/datasets/cifar.py:18: VisibleDeprecationWarning: dtype(): align should be passed as Python or NumPy boolean but got `align=0`. Did you mean to pass a tuple to create a subarray type? (Deprecated NumPy 2.4)
  d = cPickle.load(f, encoding="bytes")

50000 train samples
10000 test samples
Sample training labels shape: (50000, 1)

4. Build Heterogeneous Mixed-Precision Network Topology

x = x_in = Input(x_train.shape[1:], name="input")

x = QActivation("quantized_relu_po2(4,4)", name="acti")(x)

x = QConv2D(
    128,
    (3, 3),
    strides=1,
    kernel_quantizer=quantized_po2(4, 1),
    bias_quantizer=quantized_po2(4, 4),
    bias_range=4,
    name="conv2d_0_m",
)(x)
x = QActivation("ternary()", name="act0_m")(x)
x = MaxPooling2D(2, 2, name="mp_0")(x)

x = QConv2D(
    256,
    (3, 3),
    strides=1,
    kernel_quantizer=quantized_po2(4, 1),
    bias_quantizer=quantized_po2(4, 4),
    bias_range=4,
    name="conv2d_1_m",
)(x)
x = QActivation("quantized_relu(6,2)", name="act1_m")(x)
x = MaxPooling2D(2, 2, name="mp_1")(x)

x = QConv2D(
    128,
    (3, 3),
    strides=1,
    kernel_quantizer=quantized_bits(4, 0, 1),
    bias_quantizer=quantized_bits(4, 0, 1),
    name="conv2d_2_m",
)(x)
x = QActivation("quantized_relu(4,2)", name="act2_m")(x)
x = MaxPooling2D(2, 2, name="mp_2")(x)

x = Flatten()(x)
x = QDense(
    NB_CLASSES,
    kernel_quantizer=quantized_ulaw(4, 0, 1),
    bias_quantizer=quantized_bits(4, 0, 1),
    name="dense",
)(x)
x = Activation("softmax", name="softmax")(x)

model = Model(inputs=[x_in], outputs=[x])
model.summary()

/Users/mariuskoppel/cms/qkeras/qkeras/qconvolutional.py:274: UserWarning: bias_range is deprecated in QConv2D layer.
  warnings.warn("bias_range is deprecated in QConv2D layer.")

Model: "functional"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ input (InputLayer)              │ (None, 32, 32, 3)      │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ acti (QActivation)              │ (None, 32, 32, 3)      │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_0_m (QConv2D)            │ (None, 30, 30, 128)    │         3,584 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ act0_m (QActivation)            │ (None, 30, 30, 128)    │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ mp_0 (MaxPooling2D)             │ (None, 15, 15, 128)    │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1_m (QConv2D)            │ (None, 13, 13, 256)    │       295,168 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ act1_m (QActivation)            │ (None, 13, 13, 256)    │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ mp_1 (MaxPooling2D)             │ (None, 6, 6, 256)      │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_2_m (QConv2D)            │ (None, 4, 4, 128)      │       295,040 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ act2_m (QActivation)            │ (None, 4, 4, 128)      │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ mp_2 (MaxPooling2D)             │ (None, 2, 2, 128)      │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten)               │ (None, 512)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (QDense)                  │ (None, 10)             │         5,130 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ softmax (Activation)            │ (None, 10)             │             0 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 598,922 (2.28 MB)

 Trainable params: 598,922 (2.28 MB)

 Non-trainable params: 0 (0.00 B)

5. Compile and Fit Model Optimization

model.compile(
    loss="categorical_crossentropy", optimizer=OPTIMIZER, metrics=["accuracy"]
)

history = model.fit(
    x_train,
    y_train,
    batch_size=BATCH_SIZE,
    epochs=NB_EPOCH,
    initial_epoch=1,
    verbose=VERBOSE,
    validation_split=VALIDATION_SPLIT,
)

Epoch 2/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 55s 77ms/step - accuracy: 0.3298 - loss: 1.8858 - val_accuracy: 0.3896 - val_loss: 1.6941
Epoch 3/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 58s 83ms/step - accuracy: 0.4343 - loss: 1.5645 - val_accuracy: 0.4524 - val_loss: 1.5071
Epoch 4/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 69s 97ms/step - accuracy: 0.4857 - loss: 1.4318 - val_accuracy: 0.4820 - val_loss: 1.4327
Epoch 5/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 76s 108ms/step - accuracy: 0.5146 - loss: 1.3559 - val_accuracy: 0.5060 - val_loss: 1.3932
Epoch 6/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 68s 97ms/step - accuracy: 0.5436 - loss: 1.2827 - val_accuracy: 0.5206 - val_loss: 1.3328
Epoch 7/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 65s 93ms/step - accuracy: 0.5635 - loss: 1.2303 - val_accuracy: 0.5312 - val_loss: 1.3284
Epoch 8/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 55s 78ms/step - accuracy: 0.5852 - loss: 1.1816 - val_accuracy: 0.5540 - val_loss: 1.2644
Epoch 9/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 57s 81ms/step - accuracy: 0.6001 - loss: 1.1389 - val_accuracy: 0.5664 - val_loss: 1.2296
Epoch 10/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 63s 90ms/step - accuracy: 0.6191 - loss: 1.0888 - val_accuracy: 0.5594 - val_loss: 1.2245
Epoch 11/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 75s 106ms/step - accuracy: 0.6335 - loss: 1.0520 - val_accuracy: 0.5730 - val_loss: 1.1969
Epoch 12/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 77s 109ms/step - accuracy: 0.6481 - loss: 1.0180 - val_accuracy: 0.5680 - val_loss: 1.2212
Epoch 13/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 75s 106ms/step - accuracy: 0.6584 - loss: 0.9866 - val_accuracy: 0.5790 - val_loss: 1.2175
Epoch 14/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 71s 100ms/step - accuracy: 0.6746 - loss: 0.9462 - val_accuracy: 0.5824 - val_loss: 1.1856
Epoch 15/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 71s 102ms/step - accuracy: 0.6808 - loss: 0.9261 - val_accuracy: 0.5914 - val_loss: 1.1629
Epoch 16/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 72s 102ms/step - accuracy: 0.6924 - loss: 0.8942 - val_accuracy: 0.5980 - val_loss: 1.1367
Epoch 17/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 71s 101ms/step - accuracy: 0.7035 - loss: 0.8657 - val_accuracy: 0.5990 - val_loss: 1.1598
Epoch 18/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 65s 93ms/step - accuracy: 0.7136 - loss: 0.8420 - val_accuracy: 0.5988 - val_loss: 1.1528
Epoch 19/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 64s 92ms/step - accuracy: 0.7199 - loss: 0.8214 - val_accuracy: 0.6070 - val_loss: 1.1398
Epoch 20/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 71s 101ms/step - accuracy: 0.7317 - loss: 0.7930 - val_accuracy: 0.6134 - val_loss: 1.1286
Epoch 21/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 73s 103ms/step - accuracy: 0.7382 - loss: 0.7732 - val_accuracy: 0.6068 - val_loss: 1.1354
Epoch 22/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 74s 105ms/step - accuracy: 0.7470 - loss: 0.7506 - val_accuracy: 0.6108 - val_loss: 1.1237
Epoch 23/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 72s 103ms/step - accuracy: 0.7578 - loss: 0.7261 - val_accuracy: 0.6054 - val_loss: 1.1382
Epoch 24/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 72s 103ms/step - accuracy: 0.7596 - loss: 0.7136 - val_accuracy: 0.6114 - val_loss: 1.1223
Epoch 25/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 73s 103ms/step - accuracy: 0.7682 - loss: 0.6939 - val_accuracy: 0.6160 - val_loss: 1.1346
Epoch 26/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 70s 100ms/step - accuracy: 0.7752 - loss: 0.6786 - val_accuracy: 0.6198 - val_loss: 1.1028
Epoch 27/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 73s 103ms/step - accuracy: 0.7834 - loss: 0.6552 - val_accuracy: 0.5962 - val_loss: 1.1640
Epoch 28/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 73s 103ms/step - accuracy: 0.7898 - loss: 0.6419 - val_accuracy: 0.6140 - val_loss: 1.1412
Epoch 29/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 74s 106ms/step - accuracy: 0.7951 - loss: 0.6265 - val_accuracy: 0.6120 - val_loss: 1.1266
Epoch 30/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 79s 112ms/step - accuracy: 0.7975 - loss: 0.6179 - val_accuracy: 0.6142 - val_loss: 1.1377
Epoch 31/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 68s 97ms/step - accuracy: 0.8064 - loss: 0.5965 - val_accuracy: 0.6186 - val_loss: 1.1257
Epoch 32/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 60s 85ms/step - accuracy: 0.8124 - loss: 0.5834 - val_accuracy: 0.6122 - val_loss: 1.1492
Epoch 33/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 73s 104ms/step - accuracy: 0.8152 - loss: 0.5752 - val_accuracy: 0.6144 - val_loss: 1.1436
Epoch 34/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 74s 105ms/step - accuracy: 0.8175 - loss: 0.5642 - val_accuracy: 0.6178 - val_loss: 1.1514
Epoch 35/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 73s 104ms/step - accuracy: 0.8246 - loss: 0.5488 - val_accuracy: 0.6086 - val_loss: 1.1671
Epoch 36/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 72s 103ms/step - accuracy: 0.8246 - loss: 0.5378 - val_accuracy: 0.6110 - val_loss: 1.1405
Epoch 37/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 72s 103ms/step - accuracy: 0.8328 - loss: 0.5255 - val_accuracy: 0.6150 - val_loss: 1.1679
Epoch 38/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 73s 103ms/step - accuracy: 0.8400 - loss: 0.5015 - val_accuracy: 0.6070 - val_loss: 1.2081
Epoch 39/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 70s 100ms/step - accuracy: 0.8497 - loss: 0.4820 - val_accuracy: 0.6188 - val_loss: 1.1768
Epoch 40/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 72s 102ms/step - accuracy: 0.8506 - loss: 0.4715 - val_accuracy: 0.6084 - val_loss: 1.2076
Epoch 41/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 75s 106ms/step - accuracy: 0.8555 - loss: 0.4603 - val_accuracy: 0.6022 - val_loss: 1.2354
Epoch 42/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 75s 107ms/step - accuracy: 0.8555 - loss: 0.4501 - val_accuracy: 0.6112 - val_loss: 1.2085
Epoch 43/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 59s 84ms/step - accuracy: 0.8568 - loss: 0.4523 - val_accuracy: 0.6068 - val_loss: 1.2213
Epoch 44/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 55s 78ms/step - accuracy: 0.8586 - loss: 0.4435 - val_accuracy: 0.6068 - val_loss: 1.2183
Epoch 45/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 59s 84ms/step - accuracy: 0.8643 - loss: 0.4306 - val_accuracy: 0.6214 - val_loss: 1.2069
Epoch 46/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 55s 78ms/step - accuracy: 0.8679 - loss: 0.4204 - val_accuracy: 0.6134 - val_loss: 1.2363
Epoch 47/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 58s 82ms/step - accuracy: 0.8740 - loss: 0.4063 - val_accuracy: 0.5984 - val_loss: 1.2808
Epoch 48/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 56s 79ms/step - accuracy: 0.8758 - loss: 0.3968 - val_accuracy: 0.6112 - val_loss: 1.2256
Epoch 49/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 58s 83ms/step - accuracy: 0.8826 - loss: 0.3845 - val_accuracy: 0.6158 - val_loss: 1.2418
Epoch 50/50
704/704 ━━━━━━━━━━━━━━━━━━━━ 57s 81ms/step - accuracy: 0.8890 - loss: 0.3700 - val_accuracy: 0.6220 - val_loss: 1.2111

6. Sub-Network Target Weight Bound and Activation Profiling

outputs = []
output_names = []

for layer in model.layers:
    if layer.__class__.__name__ in [
        "QActivation",
        "Activation",
        "QDense",
        "QConv2D",
        "QDepthwiseConv2D",
    ]:
        output_names.append(layer.name)
        outputs.append(layer.output)

model_debug = Model(inputs=[x_in], outputs=outputs)
outputs = model_debug.predict(x_train)

print("{:30} {: 8.4f} {: 8.4f}".format("input", knp.min(x_train), knp.max(x_train)))

for n, p in zip(output_names, outputs):
    print(f"{n:30} {knp.min(p): 8.4f} {knp.max(p): 8.4f}", end="")
    layer = model.get_layer(n)
    for idx, weights in enumerate(layer.get_weights()):
        if layer.get_quantizers()[idx] is not None:
            quantized_tensor = layer.get_quantizers()[idx](weights)
            weights = ops.convert_to_numpy(quantized_tensor)
        print(
            f" ({knp.min(weights): 8.4f} {knp.max(weights): 8.4f})", end=""
        )
    print("")

score = model.evaluate(x_test, y_test, verbose=VERBOSE)
print("\nTest loss:", score[0])
print("Test accuracy:", score[1])

1182/1563 ━━━━━━━━━━━━━━━━━━━━ 35s 93ms/step

Der Kernel ist beim Ausführen von Code in der aktuellen Zelle oder einer vorherigen Zelle abgestürzt. 

Bitte überprüfen Sie den Code in der/den Zelle(n), um eine mögliche Fehlerursache zu identifizieren. 

Klicken Sie <a href='https://aka.ms/vscodeJupyterKernelCrash'>hier</a>, um weitere Informationen zu erhalten. 

Weitere Informationen finden Sie unter Jupyter <a href='command:jupyter.viewOutput'>Protokoll</a>.

7. Total Resource Metric Breakdown

print_qstats(model)