VISUAL RECOGNITION OF GRAPHICAL USER INTERFACE COMPONENTS USING DEEP LEARNING TECHNIQUE

VISUAL RECOGNITION OF GRAPHICAL USER INTERFACE COMPONENTS USING DEEP LEARNING TECHNIQUE

User Interface, Usability Testing, GUI, Computer Vision, Deep Learning

Graphical User Interface (GUI) building in software development is a process which ideally need to go through several steps. Those steps in the process start from idea or rough sketch of the GUI, then refined into visual design, implemented in coding or prototype, and finally evaluated for its function and usability to discover design problem and to get feedback from users. Those steps repeated until the GUI considered satisfactory. Computer vision
technique has been researched and developed to make the process faster and easier; for example generating code for implementation, or automatic GUI testing using component images. But among those techniques, there are still few for usability testing purpose. This preliminary research attempted to make the foundation for usability testing using computer vision technique by built dataset which has images of various GUI components, and used the dataset in deep learning experiment for GUI components visual recognition. The experiment results showed deep learning technique suitable for the intended task using
transfer learning as preferable method, with accuracy achieved at 95% for recognition of
two different types of component, between 80 – 94% for two similar types of component, and above 70% for six different types of GUI components.

Agyl A. Rahmadi and Aris Sudaryanto

http://dx:doi:org/10:21609/jiki:v13i1:845

Faculty of Computer Science Universitas Indonesia

2020-02-28

Sri Wahyuni

e-ISSN : 2502-9274 printed ISSN : 2088-7051

PDF

English

Text

Jurnal Ilmu Komputer dan Informasi (Journal of Computer Science and Information)