Refining the Performance of Neural Networks with Simple Architectures for Indonesian Sign Language System (SIBI) Letter Recognition Using Keypoint Detection


Nur Hikma Amir(1*); Chandra Kusuma Dewa(2); Ahmad Luthfi(3);

(1) Universitas Islam Indonesia
(2) Universitas Islam Indonesia
(3) Universitas Islam Indonesia
(*) Corresponding Author

  

Abstract


The diversity of non-verbal communication styles among persons with disabilities in Indonesia highlights the urgent need for technological solutions that support accessibility in both workplace settings and social contexts. This study proposes a novel approach to improving neural network performance through the use of simple architectures for recognizing Indonesian Sign Language (SIBI) letters M and N, by applying keypoint detection while accounting for hand size variations (17–22 cm). Four models were evaluated: YOLOv5 based on image detection, as well as VGG-16, Attention, and Multi-Layer Perceptron (MLP) developed using keypoint detection. The evaluation was conducted in real-time, taking into account accessories such as rings, watches, and gloves, as well as varying lighting intensities to simulate real-world user environments. The novelty lies in the integration of keypoint detection into lightweight architectures, which significantly improves accuracy and resilience against visual disturbances (noise). The MLP model achieved the best performance, with an accuracy of 94% for M and 93% for N, outperforming more complex approaches such as YOLOv5, which showed a significant drop in accuracy under disturbed conditions. The integration of VGG-16 with Attention resulted in underfitting, emphasizing that complexity does not always correlate with effectiveness. These findings underscore the potential of lightweight models to enhance technological accessibility for the disabled community across various social and professional domains.

Keywords


Disability; Keypoint Detection; Letter M and N; SIBI; Sign Language.

  
  

Full Text:

PDF
  

Article Metrics

Abstract view: 29 times
PDF view: 9 times 		     

Digital Object Identifier

doi  https://doi.org/10.33096/ilkom.v17i1.2522.64-73 		  

Cite

How to cite item

References


C. Suardi, A. N. Handayani, R. A. Asmara, A. P. Wibawa, L. N. Hayati, and H. Azis, “Design of Sign Language Recognition Using E-CNN,” 3rd 2021 East Indones. Conf. Comput. Inf. Technol. EIConCIT 2021, pp. 166–170, 2021, doi: 10.1109/EIConCIT50028.2021.9431877.

S. T. Abd Al-Latief, S. Yussof, A. Ahmad, S. M. Khadim, and R. A. Abdulhasan, “Instant Sign Language Recognition by WAR Strategy Algorithm Based Tuned Machine Learning,” Int. J. Networked Distrib. Comput., vol. 12, no. 2, pp. 344–361, 2024, doi: 10.1007/s44227-024-00039-8.

Cindy Mutia Annur, “Proportion of Employed Persons with Disabilities by Employment Status in Indonesia (2021-2022),” databoks, 2023.

A. Z. Yinatan, “Examining the Sectoral Distribution of Workers with Disabilities in Indonesia” Data Goodstats, 2023.

Y. Qin, S. Pan, W. Zhou, D. Pan, and Z. Li, “WiASL: American Sign Language writing recognition system using commercial WiFi devices,” Meas. J. Int. Meas. Confed., vol. 218, no. June, 2023, doi: 10.1016/j.measurement.2023.113125.

A. Sajeena, O. Sheeba, and S. S. Ajitha, “Indian sign language recognition using YOLOV5,” AIP Conf. Proc., vol. 2222, pp. 107–113, 2020, doi: 10.1063/5.0005665.

A. M. J. AL Moustafa et al., “Arabic Sign Language Recognition Systems: a Systematic Review,” Indian J. Comput. Sci. Eng., vol. 15, no. 1, pp. 1–18, 2024, doi: 10.21817/indjcse/2023/v15i1/241501008.

A. Halder and A. Tayade, “Real-time Vernacular Sign Language Recognition using MediaPipe and Machine Learning,” Int. J. Res. Publ. Rev., vol. 2, no. 5, pp. 9–17, 2021, doi: 10.13140/RG.2.2.32364.03203.

O. Sevli and N. Kemaloglu, “Turkish Sign Language digits classification with CNN using different optimizers,” Int. Adv. Res. Eng. J., vol. 4, no. 3, pp. 200–207, 2020, doi: 10.35860/iarej.700564.

M. F. Lilis Nur Hayati, Anik Nur Handayani, Wahyu Sakti Gunawan Iriantoa, Rosa Andrie Asmara, Dolly Indra, “Classifying BISINDO Alphabet using Tensorflow Object Detection API,” Ilk. J. Ilm., vol. 15, 2023, doi : 10.33096/ilkom.v15i2.1692.358-364.

E. L. R. Ewe, C. P. Lee, K. M. Lim, L. C. Kwek, and A. Alqahtani, “LAVRF: Sign language recognition via Lightweight Attentive VGG16 with Random Forest,” PLoS One, vol. 19, no. 4 April, pp. 1–22, 2024, doi: 10.1371/journal.pone.0298699.

M. C. Bagaskoro, F. Prasojo, A. N. Handayani, E. Hitipeuw, A. P. Wibawa, and Y. W. Liang, “Hand image reading approach method to Indonesian Language Signing System (SIBI) using neural network and multi layer perseptron,” Sci. Inf. Technol. Lett., vol. 4, no. 2, pp. 97–108, 2023, doi: 10.31763/sitech.v4i2.1362.

A. Alayed, “Machine Learning and Deep Learning Approaches for Arabic Sign Language Recognition: A Decade Systematic Literature Review,” Sensors, vol. 24, no. 23, 2024, doi: 10.3390/s24237798.

Tazyeen Fathima, Ashif Alam, Ashish Gangwar, Dev Kumar Khetan, and Prof. Ramya K, “Real-Time Sign Language Recognition and Translation Using Deep Learning Techniques,” Int. Res. J. Adv. Eng. Hub, vol. 2, no. 02, pp. 93–97, 2024, doi: 10.47392/irjaeh.2024.0018.

S. Dwijayanti, Hermawati, S. I. Taqiyyah, H. Hikmarika, and B. Y. Suprapto, “Indonesia Sign Language Recognition using Convolutional Neural Network,” Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 10, pp. 415–422, 2021, doi: 10.14569/IJACSA.2021.0121046.

M. Alaghband, H. R. Maghroor, and I. Garibay, “A survey on sign language literature,” Mach. Learn. with Appl., vol. 14, no. August, p. 100504, 2023, doi: 10.1016/j.mlwa.2023.100504.

I. Safar and H. Mahyar, “Hand KeyPoint Detection menggunakan Algoritma Single Shot Detector ( SSD),” J. Infomedia., vol. 9, no. 1, pp. 28–33, 2024. DOI: http://dx.doi.org/10.30811/jim.v9i1.5471.

A. Alvin, N. H. Shabrina, A. Ryo, and E. Christian, “Hand Gesture Detection for Sign Language using Neural Network with Mediapipe,” Ultim. Comput. J. Sist. Komput., vol. 13, no. 2, pp. 57–62, 2021, doi: 10.31937/sk.v13i2.2109.

S. Al Ahmadi, F. Mohammad, and H. Al Dawsari, “Efficient YOLO-Based Deep Learning Model for Arabic Sign Language Recognition,” J. Disabil. Res., vol. 3, no. 4, 2024, doi: 10.57197/jdr-2024-0051.

N. F. Attia, M. T. F. S. Ahmed, and M. A. M. Alshewimy, “Efficient deep learning models based on tension techniques for sign language recognition,” Intell. Syst. with Appl., vol. 20, no. September, p. 200284, 2023, doi: 10.1016/j.iswa.2023.200284.

S. Aiouez, A. Hamitouche, M. Belmadoui, K. Belattar, and F. Souami, “Real-time Arabic Sign Language Recognition based on YOLOv5,” SCITEPRESS, vol. 17–25, no. Improve, pp. 17–25, 2022, doi: 10.5220/0010979300003209.

T. F. Dima and M. E. Ahmed, “Using YOLOv5 Algorithm to Detect and Recognize American Sign Language,” 2021 Int. Conf. Inf. Technol. ICIT 2021 - Proc., pp. 603–607, 2021, doi: 10.1109/ICIT52682.2021.9491672.

M. A. A. K. Sanket Bankar, Tushar Kadam, Vedant Korhale, “Real Time Sign Language Recognition Using Deep Learning,” Int. J. Res. Appl. Sci. Eng. Technol., vol. 11, no. 9, pp. 193–199, 2023, doi: 10.22214/ijraset.2023.55621.

M. R. Ningsih et al., “Sign Language Detection System Using YOLOv5 Algorithm to Promote Communication Equality People with Disabilities,” Sci. J. Informatics, vol. 11, no. 2, pp. 549–558, 2024, doi: 10.15294/sji.v11i2.6007.

S. Feng, L. Zhao, H. Shi, M. Wang, S. Shen, and W. Wang, “One-dimensional VGGNet for high-dimensional data,” Appl. Soft Comput., vol. 135, p. 110035, 2023, doi: 10.1016/j.asoc.2023.110035.

C. Suardi, “CNN architecture based on VGG16 model for SIBI sign language,” ETLTC-ICETM2023 Int. Conf. Proc. ICT Integr. Tech. Educ. Entertain. Technol. Manag., vol. 2909, no. 120010, 2023, doi: 10.1063/5.0181956.

M. A. Rajab, F. A. Abdullatif, and T. Sutikno, “Classification of grapevine leaves images using VGG-16 and VGG-19 deep learning nets,” Telkomnika (Telecommunication Comput. Electron. Control., vol. 22, no. 2, pp. 445–453, 2024, doi: 10.12928/TELKOMNIKA.v22i2.25840.

T. N. A.-J. S. S. Abu-Naser, “Classification of Sign language Using VGG16,” ISAS 2023 - 7th Int. Symp. Innov. Approaches Smart Technol. Proc., vol. 6, no. 6, pp. 36–46, 2022, doi: 10.1109/ISAS60782.2023.10391673.

C. J. K. Priyanka, R. Rithik, “A Fusion of CNN, MLP, and MediaPipe for Advanced Hand Gesture Recognition,” Int. Conf. Recent Adv. Sci. Eng. Technol. IEEE, vol. 1–5, 2024, doi : 10.1109/ICRASET63057.2024.10895921.

B. Joksimoski et al., “Technological Solutions for Sign Language Recognition: A Scoping Review of Research Trends, Challenges, and Opportunities,” IEEE Access, vol. 10, pp. 40979–40998, 2022, doi: 10.1109/ACCESS.2022.3161440.

Y. Saleh and G. F. Issa, “Arabic sign language recognition through deep neural networks fine-tuning,” Int. J. online Biomed. Eng., vol. 16, no. 5, pp. 71–83, 2020, doi: 10.3991/IJOE.V16I05.13087.

S. Kumar, R. Rani, and U. Chaudhari, “Real-time sign language detection: Empowering the disabled community,” MethodsX, vol. 13, no. August, p. 102901, 2024, doi: 10.1016/j.mex.2024.102901.

S. Lyu, “SC2SLR: Skeleton-based Contrast for Sign Language Recognition,” Proc. 2024 5th Int. Conf. Comput. Networks Internet Things, vol. 4040410, 2024, doi : 10.1145/3670105.367017.

S. M. K. Raja’a M. Mohammed, “Iraqi Sign Language Translator system using Deep Learning,” Al-Salam J. Eng. Technol., vol. 1, pp. 109–116, 2023, doi: 10.55145/ajest.2023.01.01.0013.


Refbacks

  • There are currently no refbacks.


Copyright (c) 2025 Nur Hikmah Amir, Chandra Kusuma Dewa

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.