A Single Graph Convolution is All You Need: Efficient Grayscale Image Classification (2024)

1 Introduction

As the demand and popularity of real-time systems increase, low-latency machine learning has become increasingly important. With more and more consumers interacting with machine learning models through the cloud, the speed at which those models can deliver results is critical. Consumers expect fast and accurate results; any latency can lead to a poor user experience. Moreover, low-latency machine learning is essential in real-time applications, such as autonomous vehicles or stock market trading, where decisions must be made quickly and accurately. In these scenarios, delays caused by high latency can result in severe consequences and even cause inaccurate downstream calculations[1].

A particular instance where low-latency machine learning is needed is grayscale image classification for SAR ATR. For example, a targeting system on a satellite is costly, and decisions must be made using SAR efficiently and accurately. Examples like this are where low-latency grayscale image classification comes into play. It is often the case that image classifiers work on RGB datasets and grayscale image datasets, but seldom do modern image classifiers focus solely on the grayscale setting. RGB models are overkill for the grayscale setting, as the grayscale problem allows us to focus on a single channel. Models focusing on grayscale image classification are naturally more efficient, as they can concentrate on a single channel rather than three. Thus, many image classifiers that generalize to the grayscale image classification are not truly optimized for the grayscale case. For these reasons, we present a lightweight grayscale image classifier capable of achieving up to $16\times$ lower latency than other state-of-the-art machine learning models on the MSTAR dataset.

From a trustworthy visual data processing perspective, the demand for grayscale image classification requires data to be collected from various domains with high resolution and correctness so that we can train a robust machine learning model. Additionally, recent advancements in machine learning rely on convolutional neural networks, which often suffer from high computation costs, large memory requirements, and many computations needed, resulting in poor inference latency, poor scalability, and weak trustworthiness.

The inherent novelties of our model are as follows: Our proposed method is the first to vectorize an image in a fully connected manner and input the resultant into a single-layer graph convolutional network (GCN). We also find that a single GCN layer is enough to stabilize the performance of our shallow model. Additionally, our proposed method benefits from a batch-wise attention term, allowing our shallow model to capture interdependencies between images and form connections for classification. Finally, by focusing on grayscale imagery, we can focus on a streamlined method for grayscale image classification rather than concentrating on the RGB setting. A result of these novelties is extremely low latency and high throughput for SAR ATR and medical image classification.

2 Problem Definition

The problem is to design a lightweight system capable of handling high volumes of data with low latency. The solution should be optimized for performance and scalability while minimizing resource utilization, a necessary component of many real-time machine learning applications. The system should be able to process and respond to requests quickly, with minimal delays. High throughput and low latency are critical requirements for this system, which must handle many concurrent requests without compromising performance. We define latency and throughput in the following ways:

Latency refers to the total time (from start to finish) it takes to gather predictions for a model in one batch. A lightweight machine learning model aims to maximize throughput and accuracy while minimizing latency.

3 Related Work

4 Overview and Architecture

This section describes our model architecture (GECCO: Grayscale Efficient Channelwise Classification Operative). The overall process is summarized in Figure1.

Overall Architecture. Many existing methods do not focus on the latency of their design and its implications. Additionally, the vast majority of image classification models focus on the performance of their work in the RGB setting, rarely citing the performance of datasets in various domains. We address these problems by presenting a novel architecture focused on low latency and the grayscale image setting.

Our model vectorizes a batch of images, allowing us to use a fully connected layer (FC) pixel-wise for low computation time rather than relying on convolutional neural networks. We vectorize the input images and input them into a fully connected layer. Then, we use a graph convolutional layer to learn similarities between images batch-wise. We then apply a batch-wise attention term, which is inputted into an FC for classification.¹¹1We make our code publicly available at https://github.com/GECCOProject/GECCO

Image Vectorization. For each image in a batch, we view the image as a vector. For a tensor $\mathbf{X}\in\mathbb{R}^{B\times H\times W}$ where $B$ is the batch size,$H$ and $W$ are the height and width of an image; we flatten the tensor to $\mathbf{X}_{1}\in\mathbb{R}^{B\times(H\cdot W)}$.Viewing an image as a vector allows our model to skip the traditional convolutional neural network, which views the image as a grid and cuts computation time.

Fully Connected Layer. We input $\mathbf{X}_{1}$ into an FC layer with output dimensionality $D_{out}$.Formally, $\mathbf{X}_{2}=\sigma(\mathbf{X}_{1}\mathbf{W}_{1}+\mathbf{b}_{1})$, where $\sigma$ is the ReLU function,$\mathbf{W}_{1}$ is a learned weight matrix, $\mathbf{b}_{1}$ is a bias term, and $\mathbf{X}_{2}\in\mathbb{R}^{B\times D_{out}}$.

After the fully connected layer, we apply a dropout layer and the ReLU function to $\mathbf{X}_{2}$, yielding $\mathbf{X}_{3}$,such that the resultant dimensionality of $\mathbf{X}_{3}$ is $\mathbb{R}^{B\times D_{out}}$.

Graph Construction. We construct a graph batch-wise from $\mathbf{X}_{3}$. This means that for each batch, a vectorized image is each node in the graph with feature size$\mathbb{R}^{D_{out}}$, and each image is connected to every other image in a batch. Formally, we calculate the adjacency matrix$\mathbf{A}$ as $\mathbf{A}_{ij}=1$, which connects all nodes.

Graph Convolution. Our single graph convolutional layer learns from similar features of images within its minibatch.Generally, a graph convolutional layer updates the representations of nodes by aggregating each node’s neighbor’s representation.We can write a graph convolutional layer as $\bm{h^{\prime}}_{i}=f_{\theta}\left(\bm{h}_{i},\text{\scriptsize{AGGREGATE}}$.In our case, the input for each node $\bm{h_{i}}$ is the output from each vector in $\mathbf{X}_{3}$.

Applying graph convolution to $\mathbf{X}_{3}$ yields $\mathbf{X}_{4}$.Formally, $\mathbf{X}_{4}=\sigma\left(\mathbf{A}\mathbf{X}_{3}\mathbf{W}_{2}\right)$, where $\mathbf{W}_{2}$ is a learned weight matrix and $\sigma$ represents the sigmoid function.After the graph convolution, we apply batch normalization and max-pooling operations to $\mathbf{X}_{4}$, resulting in adimensionality of $\mathbb{R}^{B\times\lfloor D_{out}/2\rfloor}$.

The residual connection is defined $\mathbf{X}_{6}=\mathbf{X}_{5}+\mathbf{X}_{4}$. The residual term makes the learning process easier and more stable.By multiplying a softmax-like term with the output of the previous graph convolution,we weigh the correspondence of each image compared to other similar images within similar images batch-wise. We then feed the residual term into an FC inputted into the softmax function for classification results.

5 Experiments

5.2 Results

5.2.1 Backbone

For Table1, we choose ResConv as the backbone of our model because it has the most desirable characteristics for applying a graph convolutional layer.

Convolutional Layer	Top-1 Accuracy	Throughput ($\text{imgs}/\text{ms}$)	Latency (ms)
GCN[13]	$98.89\%$	$50.04\pm 6.85$	$5.86\pm 0.98$
TAGConv[14]	$99.05\%$	$47.87\pm 7.11$	$6.24\pm 1.32$
SAGEConv[15]	$99.08\%$	$51.77\pm 8.19$	$5.95\pm 0.87$
ChebConv[16]	$98.56\%$	$45.37\pm 5.99$	$6.83\pm 1.27$
ResConv[17]	$\bf{99.29\%}$	$\bf{52.98\pm 9.04}$	$\bf{5.22\pm 1.03}$

We use the following hyperparameters listed in Table 2 for our experiments.

Dataset	Feature Length	Optimizer	Batch Size
MSTAR	$86$	Adam	$64$
CXR	$112$	Adam	$64$

5.2.2 Experimental Performance

Experimental performance includes the top-1 accuracy, inference throughput, and inference latency. We perform our inference batch-wise as a means to reduce latency. These metrics vary across each dataset.

We summarize our findings in Tables3 and 4. We report the best-performing accuracy, average throughputs, and latencies with their standard deviations. Our model outperforms every other model in terms of throughput and latency across all datasets, leads accuracy on the MSTAR dataset, and performs competitively in terms of accuracy on all datasets.

We perform the remaining experiments on a state-of-the-art NVIDIA RTX A5000 GPU. Additionally, we compare our model to the top-performing variants of VGG[18], the variant of the popular ViT[19], the ViT for small-sized datasets (SS-ViT)[20], FastViT[21], Swin Transformer[22], and ResNet[23] models. We use the open-source packages PyTorch and HuggingFace for model building and the PyTorch Op-Counter for operation counting. Performing the remaining experiments on the same hardware system is vital in fostering a fair comparison for each model.

Model	Top-1 Accuracy	Throughput ($\text{imgs}/\text{ms}$)	Latency (ms)
Swin-T	$86.04\%$	$1.36\pm 0.10$	$46.98\pm 3.20$
SS-ViT	$95.61\%$	$2.29\pm 0.43$	$27.97\pm 5.26$
VGG16	$93.13\%$	$1.69\pm 0.33$	$37.89\pm 7.52$
FastViT	$91.78\%$	$1.04\pm 0.13$	$61.44\pm 7.69$
ResNet34	$98.64\%$	$3.13\pm 0.22$	$20.48\pm 1.39$
GECCO	$\bf{99.29\%}$	$\bf{12.26\pm 2.42}$	$\bf{5.22\pm 1.03}$

Model	Top-1 Accuracy	Throughput ($\text{imgs}/\text{ms}$)	Latency (ms)
Swin-T	$73.66\%$	$0.27\pm 0.05$	$236.71\pm 46.09$
SS-ViT	$71.09\%$	$1.03\pm 0.21$	$62.35\pm 12.85$
VGG16	$\bf{82.01\%}$	$0.76\pm 0.25$	$84.10\pm 28.43$
FastViT	$75.46\%$	$1.06\pm 0.14$	$60.30\pm 14.24$
ResNet34	$78.31\%$	$0.60\pm 0.11$	$105.84\pm 19.39$
GECCO	$77.57\%$	$\bf{2.63\pm 0.55}$	$\bf{24.32\pm 5.08}$

5.2.3 Model Complexity Metrics

Model complexity metrics for this paper include the number of multiply-accumulate operations (MACs), the number of model parameters, the model size, and the number of layers. In other words, suppose accumulator $a$ counts an operation of arbitrary $b,c\in\mathbb{R}$. We count the number of multiply-accumulate operations as $a\leftarrow a+(b\times c)$. Additionally, the layer count metric is an essential factor of latency. Decreasing the number of layers will also improve the latency of a model’s inference time. The goal of an effective machine learning model is to maximize throughput while minimizing the number of MACs and the number of layers, in our case.

We measure the model complexity of our model against other popular machine learning models that we have chosen in Table5. Our model outperforms in all categories regarding our chosen model complexity metrics, highlighting its lightweightness.

Model	# MACs	# Parameters	Model Size (Mb)	# Layers
Swin-T	$2.12\times 10^{10}$	$2.75\times 10^{7}$	$109.9$	$167$
SS-ViT	$1.55\times 10^{10}$	$4.85\times 10^{6}$	$19.62$	$79$
VGG16	$9.51\times 10^{9}$	$4.69\times 10^{6}$	$18.75$	$20$
FastViT	$7.16\times 10^{8}$	$4.02\times 10^{6}$	$16.1$	$226$
ResNet34	$4.47\times 10^{9}$	$2.13\times 10^{7}$	$85.1$	$92$
GECCO	$\bf{5.10\times 10^{4}}$	$\bf{5.08\times 10^{4}}$	$\bf{0.19}$	$\bf{16}$

5.2.4 Ablation Study

We perform an ablation study to verify that the components of our proposed model contribute positively to the overall accuracy of the MSTAR dataset.

Mini-batch GNN	Weighted Sum Residual Term	Accuracy on MSTAR
✓	✓	$99.29\%$
✗	✓	$97.94\%$
✓	✗	$88.04\%$
✗	✗	$78.64\%$

Additionally, we find that only a single graph convolutional layer is enough to reduce the variance and increase the accuracy of our model. Refer to Figure2.

5.4 FPGA Implementation

We develop an accelerator for the proposed model on state-of-the-art FPGA, Xilinx Alveo U200[24], to further highlight the model’s efficiency and compatibility with hardware. It has 3 Super Logic Regions (SLRs), 4 DDR memory banks, 1182k Look-up tables, 6840 DSPs, 75.9 Mb of BRAM, and 270 Mb of URAM. The FPGA kernels are developed using the Xilinx High-level Synthesis (HLS) tool to expedite the design process.

Our FPGA design incorporates several novel features: (1) Portability of the design: We design a parameterized hardware template using HLS. It is portable to different FPGA platforms, including embedded and data-center FPGAs. We present our hardware mapping algorithm in Algorithm 1. (2) Resource sharing: The model is executed layer-by-layer. Each layer in the model is decomposed into basic kernel functions. The basic kernel functions, including matrix multiplication, elementwise activation, column-wise and row-wise summations, max pooling, and various other elementwise operations, are implemented separately and subsequently invoked within their corresponding layers. Due to the reuse of these fundamental kernel functions across multiple layers, FPGA resources are shared among the different layers, maximizing runtime hardware resource utilization. (3) Single-load strategy: We employ a one-time data load strategy to load the required data from DDR only once. All other data required for the computations are stored in on-chip memory, reducing inference latency. Figure3 illustrates the overall hardware architecture of our design.

We utilize the Vitis tool[25] for hardware synthesis and place-and-route to determine the achieved frequency. The Vitis Analyzer tool is then used to report resource utilization and the number of clock cycles. The latency is calculated by multiplying the achieved clock period by the number of cycles. Table8 reports the results obtained for the MSTAR dataset. Given the compact design and resource efficiency of the model, it can be accommodated within a single SLR. Hence, we deploy multiple accelerator instances across multiple SLRs, each with one instance. This increases the inference throughput. Table8 shows the latency obtained for a single inference and the throughput achieved by running the design on 3 SLRs concurrently.

GPUOur DesignPlatformNVIDIA A5000Alveo U200TechnologySamsung 8 nmTSMC 16 nmFrequency1.17 GHz200 MHzPeak Performance (TFLOPS)27.70.66On-chip Memory6 MB35 MBMemory Bandwidth768 GB/s77 GB/sLatency on MSTAR (ms)5.225.65Throughput on MSTAR (imgs/ms)12.2633.98

1:Model $f()$ and the input images;

2:Execution result;

3:foreach layer $i$ in $f()$do

4:if layer $i$ is a fully connected layer then

5:Map to the matrix multiplication unit

6:iflayer $i$ is a graph convolution layerthen

7:Map to the matrix multiplication unit

8:Map to the activation unit

9:Map to the elementwise operation unit

10:Map to the matrix addition unit

11:iflayer $i$ is a batch-wise attentionthen

12:Map to the matrix multiplication unit

13:Map to the activation unit

14:Map to the elementwise operation unit

15:iflayer $i$ is a max pooling layerthen

16:Map to the max pooling unit

17:iflayer $i$ is a activation layerthen

18:Map to the activation unit

19:iflayer $i$ is a batch normalization layer then

20:Map to the batch normalization unit

We compare our FPGA implementation with the baseline GPU implementation. The GPU baseline is executed on an NVIDIA RTX A5000 GPU, which operates at 1170 MHz and has a memory bandwidth of 768 GB/s. However, the FPGA operates at 200 MHz and has an external memory bandwidth of 77 GB/s. We compare the hardware features of the two platforms in Table7. Although the GPU has higher peak performance ($41\times$) and memory bandwidth ($10\times$), our FPGA implementation achieves a comparable latency of $5.65~{}ms$ and an improved throughput of $33.98~{}imgs/ms$.

Latency	$5.65$ ms
Throughput	$33.98$ imgs/ms
BRAMs	$956~{}(22\%)$
URAMs	$228~{}(24\%)$
DSPs	$1226~{}(17\%)$
LUTs	$459K~{}(38\%)$
FFs	$597K~{}(25\%)$

6 Conclusion and Future Work

This work introduced a novel architecture combining fully connected and graph convolutional layers, benchmarked on popular grayscale image datasets. The model demonstrated strong performance and low complexity, highlighting the importance of lightweight, low-latency image classifiers for various applications. Its efficacy was shown across SAR ATR and medical image classification, with an FPGA implementation underscoring its hardware friendliness. Key innovations include using a single-layer GCN, which, along with batch-wise attention, enhances accuracy and reduces variance. Future work should explore extending this approach to color image datasets and other domains, optimizing the architecture for even greater efficiency, and further investigating the potential of graph neural networks in shallow models.

7 Acknowledgement

This work is supported by the DEVCOM Army Research Lab (ARL)under grant W911NF2220159. Distribution Statement A: Approved for public release. Distribution is unlimited.

References

• [1]Kaoru Ota, MinhSon Dao, Vasileios Mezaris, and Francesco G. B.De Natale,“Deep learning for mobile multimedia: A survey,”ACM Trans. Multimedia Comput. Commun. Appl., vol. 13, no. 3s,jun 2017.
• [2]Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, AlaaeldinEl-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve,Jakob Verbeek, and Hervé Jégou,“Resmlp: Feedforward networks for image classification withdata-efficient training,” 2021.
• [3]Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai,Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, JakobUszkoreit, Mario Lucic, and Alexey Dosovitskiy,“Mlp-mixer: An all-mlp architecture for vision,” 2021.
• [4]Benjamin Sanchez-Lengeling, Emily Reif, Adam Pearce, and AlexanderB.Wiltschko,“A gentle introduction to graph neural networks,”Distill, 2021,https://distill.pub/2021/gnn-intro.
• [5]Naman Goyal and David Steiner,“Graph neural networks for image classification and reinforcementlearning using graph representations,” 2022.
• [6]Bingyi Zhang, Rajgopal Kannan, Viktor Prasanna, and Carl Busart,“Accurate, low-latency, efficient sar automatic target recognitionon fpga,”in 2022 32nd International Conference on Field-ProgrammableLogic and Applications (FPL). Aug. 2022, IEEE.
• [7]Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu,“Vision gnn: An image is worth graph of nodes,”in NeurIPS, 2022.
• [8]ArnabKumar Mondal, Vineet Jain, and Kaleem Siddiqi,“Mini-batch graphs for robust image classification,” 2021.
• [9]Qishang Cheng, Hongliang Li, Qingbo Wu, and KingNgi Ngan,“Ba2m: A batch aware attention module for image classification,”2021.
• [10]Yaochen Hu, Amit Levi, Ishaan Kumar, Yingxue Zhang, and Mark Coates,“On batch-size selection for stochastic training for graph neuralnetworks,” 2021.
• [11]Shuzhi Yu and Carlo Tomasi,“Identity connections in residual nets improve noise stability,”2019.
• [12]Daniel Kermany,“Labeled optical coherence tomography (oct) and chest x-ray imagesfor classification,” 2018.
• [13]ThomasN. Kipf and Max Welling,“Semi-supervised classification with graph convolutional networks,”2017.
• [14]Jian Du, Shanghang Zhang, Guanhang Wu, Jose M.F. Moura, and Soummya Kar,“Topology adaptive graph convolutional networks,” 2018.
• [15]WilliamL. Hamilton, Rex Ying, and Jure Leskovec,“Inductive representation learning on large graphs,” 2018.
• [16]Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst,“Convolutional neural networks on graphs with fast localizedspectral filtering,” 2017.
• [17]Xavier Bresson and Thomas Laurent,“Residual gated graph convnets,” 2018.
• [18]Karen Simonyan and Andrew Zisserman,“Very deep convolutional networks for large-scale imagerecognition,” 2015.
• [19]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, GeorgHeigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby,“An image is worth 16x16 words: Transformers for image recognitionat scale,” 2021.
• [20]SeungHoon Lee, Seunghyun Lee, and ByungCheol Song,“Vision transformer for small-size datasets,”CoRR, vol. abs/2112.13492, 2021.
• [21]Pavan KumarAnasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and AnuragRanjan,“Fastvit: A fast hybrid vision transformer using structuralreparameterization,” 2023.
• [22]ZeLiu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, andBaining Guo,“Swin transformer: Hierarchical vision transformer using shiftedwindows,” 2021.
• [23]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,“Deep residual learning for image recognition,”in 2016 IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2016, pp. 770–778.
• [24]Xilinx,“Xilinx alveo u200 board,”https://docs.xilinx.com/r/en-US/ds962-u200-u250/FPGA-Resource-Information.
• [25]“Vitis HLS,”https://www.xilinx.com/products/design-tools/vitis/vitis-hls.html.