Enhancement of Human Sensory and Motor Skills using Collaboration of Eye-Ear Focus and Body-Arm-Hand Gesture in No-Touch 3D Interactive Aircraft Game

Nowadays, human sensory and motor skills enhancement is very important to be alert from the effect of the Society 5.0 era. We propose the novel approach to enhance human sensory and motor skills using collaboration of senses focus and body parts gesture in no-touch 3D interactive aircraft game. A player interacts with a virtual display using gestures of his body, arm, hand, or collaboration of them as a controller to engage in a 3D game. This game is based on an aircraft education theme. By sweeping the hand to the left or right, a player can select one of many different aircraft models according to its specification and performance are shown on the screen based on the estimation of his current skill. The mission is to control aircraft avoiding the random balls attack to prevent a collision. Hand and arm are positioned above a leap motion sensor and in front of a camera. The body, arm, and hand gesture directions are to forward or backward from the screen to the right or left, and also to up or down freely. This system is developed by using Unity 3D, OpenCV, and Python. Many skills are possible to be improved, e.g., motivation, discipline, safety awareness, endurance management, strategy, precision, intelligence, creativity, reliability, toughness, and vision. This project has been tested in the Air Force Museum of Dirgantara Mandala involving more than 1000 visitors as testing participants, and they all feel satisfied. The score is calculated by time in second. The results show endurance time is 32% (<60 s), 31% (<120 s), 25% (<160 s), and 12 % (180 s) of the total 300 seconds of the game.


Introduction
Currently, the world is in the era of the Industrial Revolution 4.0 where all activities of people's lives rely on Information and Communication Technology (ICT). The progress of civilization is increasing with the emergence of Society 5.0 which forces humans to become part of the technology itself. Various applications of Society 5.0 affect several important aspects, including sensory and motor abilities, which ultimately affect the individual's ability to work. This condition for all areas of life requires special attention, especially those related to the tasks it carries out because of the consequences of the Society 5.0 era which is running and growing rapidly. Humans need to improve themselves to be alert in this era of the Industrial Revolution 4.0. Nowadays, human sensory and motor skills enhancement is very important to be alert from the effect of the Society 5.0 era.
In the current era of Society 5.0, many human activities are influenced by the use of gadgets and computers to complete various goals in life. Various activities using gadgets and computers require a lot of concentration, combining hand movements (including fingers) with the five senses, namely eyes, ears, mouth, etc. The movement between humans in communicating and interacting physically is decreasing because it is caused by the ease of information technology that is channeled in these gadgets and computers. In addition to the benefits, the impact of using gadgets for humans also exists, for example, radiation on the body and eyes [1] and [2]. This has the potential to affect human sensory and motor skills.
The dynamics of the community's mindset that is built based on the 4.0 industrial revolution is strongly influenced by technology. For this reason, trust, positive acknowledgment, and interest in a task in an organization that always prioritizes safety (for example in aviation, mining, and the military) require solutions that are by the industrial revolution 4.0. The solution taken must be able to improve sensory and motor skills using appropriate but efficient methods. Leap Motion is a sensor device in the form of hardware used with computers, produced by Leap Motion Inc. Leap Motion accepts finger and hand movements as input, similar to a mouse but does not require hand contact or touch [3]. Leap Motion Inc. launched new software in 2016 which is used to track hands in Virtual Reality. The depiction of the field of view in Leap Motion is as illustrated in Figure 1. Therefore, in this study, we design and develop a Leap Motion-based game to improve human sensory and motor skills. The emphasis of the Aero Gesture game introduced in this study is a game with the theme of controlling several aircraft models that have been used during the historical era of the Indonesian war of independence

Related Works
To support this research, several previous studies have been reviewed. Several previous studies have reported on the use of interactive games to support the rehabilitation of several things. Rodríguez-Fórtiz, et al. [4] in 2015 detailed the goals of the VIRTRA-EL web platform, which has been designed to evaluate and train cognitive skills to elderly users through the use of personal computers or tablet devices. In addition, they present serious games based on interactive 3D environments, which have been designed to help to train memory, attention, planning, and reasoning. Ling and Rui [5] in 2016 tried VR glasses and Leap Motion in conjunction with the two objects, optimizing the combination of these two products to play their strengths, and apply them to educate them, to improve teaching efficiency, to improve student learning efficiency, so education it can be optimized. The main research methods used are library research and case analysis. Pambudi, et al. [6] in 2016 proposed a psychomotor development game using Leap Motion Technology. This technology can interact with 3D space. A skeletal tracking method to identify and detect bones and fingers in human hands will be applied. This study has an accuracy rate of 96.5% for detecting the human hand with the Leap Motion device. This technology overcomes the noise caused by sunlight. The tolerance distance between the user's hands is 20 to 50 cm above the Leap Motion device.
Alimanova, et al. [7] in 2017 presented a game for hand rehabilitation using a Leap Motion controller. The main idea of gamified hand rehabilitation is to help develop muscle tone and improve accuracy in movement using the opportunities that VR offers by making the rehabilitation process more effective and motivating the patient. Yıldız, et al. [8] in 2018 developed a physiotherapy game application with the Leap Motion device and the Unity motor game so that physiotherapy exercises can be carried out properly to overcome the difficulty of finger and joint movement of stroke patients.
Shah, et al. [9] in 2019 presented a system to provide automatic and non-invasive fine motor rehabilitation through rhythm-based play using the Leap Motion Controller. The platform is a rhythm game where hand movements are used as input and must match the rhythms and movements displayed on the screen, thus allowing the physical therapist to represent the training session as a series of patterns involving the joints of the user's hand and fingers. Fine motor rehabilitation plays an important role in the recovery and amelioration of the effects of diseases and conditions such as stroke, Parkinson's disease, multiple sclerosis, among many others. Individuals with this condition have various disorders such as fine motor movements. The proposed serious game is adaptive for players to allow access to patients of various abilities.
Dhanalaksmi, et al. [10] 2020 introduced a new game based on augmented reality called Athynos. Hardware components such as sensors (ultrasonic sensors) are used to deliver virtuality in the Athynos game through the air selection process. The hand movement of the dyspraxia child is captured by the ultrasound sensor and using Arduino the captured action is transferred to the system as input. As a result, dyspraxia children were analyzed with three or more trials and the reports were calculated using a deep learning algorithm (linear regression). Active gameplay (Athynos) is used as a psychotherapy tool for children with dyspraxia. Athynos Games uses augmented reality to help improve skills that develop cognitive learning and hand-eye synchronization for children with dyspraxia.

Proposed Model
We propose the novel approach to enhance human sensory and motor skills using collaboration of senses focus and body parts gesture in a no-touch 3D interactive aircraft game called Aero Gesture [11]. There are several types of sensory according to experts.
[12] The 5 basic sensory systems are Visual, Auditory, Olfactory (smell) System, Gustatory (taste) System, and Tactile System. The 3 sensory systems Ayres focused on in describing sensory integration dysfunction areTactile System, Vestibular (sense of head movement in space) System, and Proprioceptive (sensations from muscles and joints of the body) System. The most recently discussed set of sensations related to internal organs is inter conception. A player interacts with a virtual display using gestures of his body, arm, hand, or collaboration of them as a controller to engage in a 3D game.
The method used in making the game uses Unity 3D, by prioritizing historical objects that are packaged in a pattern to be an airplane controller. Some targets regarding attitudes and characters are deliberately determined so that they affect the passion in making this game. Players are introduced to the history, profile, and specifications of the selected aircraft. This stage is deliberately carried out and becomes an integral part of the purpose of making this game, which is to improve sensory and motor skills along with increasing education about the history of the country (Figure 2). This game is based on an aircraft education theme. By sweeping the hand to the left or right, a player can select one of many different aircraft models according to its specification and performance are shown on the screen based on the estimation of his current skill. The mission is to control aircraft avoiding the random balls attack to prevent a collision. Hand and arm are positioned above a leap motion sensor and in front of a camera. The body, arm, and hand gesture directions are to forward or backward from the screen to the right or left, and also to up or down freely. This system is developed by using Unity 3D, OpenCV, and Python.

Step 1, Arena Preparation and Game Console Development
Arena Preparation and Game Console Device Development. With the following goals: • Availability of arenas to place interactive game consoles.
• Availability of three packages of Aero Gesture game console hardware.
• Availability of ready-to-play Aero Gesture game applications (software).

3.2.
Step 1, Testing and Implementation for Museum visitors Testing and Implementation for Museum visitors (Period I: January 1 to March 31, 2020; Period II: April 1 to October 31, 2020), with the following targets: • Integrated game console devices (hardware), motion sensors, and Aero Gesture game applications. • Tested interactive and educational gameplay Aero Gesture.
• The implementation of interactive and educational games for Aero Gesture TNI AU to the public visiting the Mandala Dirgantara Museum. • The implementation of convenience, comfort, and fun for the players of the Indonesian Air Force Aero Gesture game using the usability testing method.

3.3.
Step 3, Testing the memory response of visitors/players of Aero Gesture games related to the Indonesian Air Force with the following targets: Testing the memory response of visitors/players of Aero Gesture games related to the Indonesian Air Force with the following targets: • The achievement of an experiment that explores the memory of Aero Gesture game players which is associated with knowledge of the Indonesian Air Force. Step 4. Testing the Educational Principles of Aero Gesture Interactive Games with the following targets: Testing the Educational Principles of Aero Gesture Interactive Games with the following targets: • The achievement of experiments on the impact of educational principles on the community of players of the Aero Gesture interactive game. • The implementation of an interactive Aero Gesture Game playing competition for school and military students/adolescents with a reference to the gradual and integrated increase in the duration of achievement in playing the game. 3.5.
Step 5, Evaluation and Report Generation.
• Evaluated the implementation of Aero Gesture interactive and educational games by integrating game console devices (hardware), motion sensors, and Aero Gesture game applications in an integrated manner using Artificial Intelligence, forward chaining, and backward chaining methods. • Completion of reports on the implementation of research and creation of Aero Gesture interactive game innovations

How to play This Aero
Gesture interactive game is played using a certain attitude/gesture movement sensor, without sticks or other buttons. Aero Gesture is played with a palm gesture that is swung/moved around the sensor. The Aero Gesture game console is shown in Figure 3. The situation when data collection for Aero Gesture game players is shown in Figure 4. There are two stages in playing Aero Gesture. Tupolev, Glider Kampret, and Nishikoreng. To select which plane to fly, the player swings his palm over the sensor to the left or right. • The Second stage, to select the desired plane, the player swings his palm from a raised position above the sensor in an upward direction. Furthermore, players will take on the role of a pilot who controls the plane by interacting virtually using palm movements that represent the plane's position. The palm plane can be moved freely in any direction to control the plane whose job it is to avoid a lot of asteroid rock balls. An indication of the success of the Aero Gesture is the time used in seconds.   The data on each aircraft displayed strengthens players' memory of the role of the Indonesian Air Force for the post-independence struggle in Indonesia. Players who act as pilots who use their hand swings when playing will feel they have the plane and feel as if they were in an era where these historic aircraft played a role in the history of the Indonesian nation. Figure 7. Score display (yellow dotted circle) that indicates the endurance ability of game players Many skills are possible to be improved, e.g. motivation, discipline, safety awareness, endurance management, strategy, precision, intelligence, creativity, reliability, toughness, and vision. This project has been tested in the Air Force Museum of Dirgantara Mandala involving more than 1000 visitors as testing participants, and they all feel satisfied. The score is calculated by time in second (See Figure 7). The results show endurance time is 32% of 1000 players (<60 s), 31% (<120 s), 25% (<160 s), and 12 % (180 s) of total 300 seconds of the game (first stage).

Character Building Aspects
There are many character-building aspects in this game. When playing the game, players feel like a drone pilots because they control the plane by using energetic swings of hand gestures. The educational aspect of the interactive Aero Gesture game is designed with the following principles: Precision, Strategy, Safety, Tenacity, Fighting, Endurance, Creativity, Intelligence, Skilled, Agile, Tough, Reliable, Visionary, Innovative, Responsive, Speed, Patience, Confidence, Courage, Accuracy, Energetic, Quick Decision, Educational, Focus, Kinesthetic, and Happiness. Table 1 below is an illustration of the relationship between aspects of character building and the five main factors of Aero Gesture, namely Eyes, Ears, Body Gestures, Hand Gestures, and Arm Gestures. The Aero Gesture game is designed as a normalization of the habits of the players who have been actively playing games/activities on gadgets. In contrast to games that use sticks or gadgets, Aero Gesture games are specifically designed to quickly return the player's senses of sight and hearing to the brain, which is then realized by making decisions to move the body responsively. This action becomes a cycle that repeats quickly and can improve memory, quick response, and quick decision making, but the feedback is a combination of energetic sensory and motor roles in the body according to the principles mentioned above.

Conclusions
Nowadays, human sensory and motor skills enhancement is very important to be alert from the effect of the Society 5.0 era. Aero Gesture Game is an interactive and educative game that displays the game of historic aircraft in the Indonesian Air Force which is located at the Central Museum of the Indonesian Air Force Dirgantara Mandala (Muspusdirla) Yogyakarta. The Aero Gesture game invites players to take on the role of pilots of the aircraft of their choice by using the swing of their palms. This interactive game is to enhance Human Sensory and Motor Skills using the Collaboration of Eye -Ear Focus and Body -Arm -Hand Gesture in No-Touch 3D Interactive Aircraft Game. Future researches that are expected include the development of a system for calculating sensory and motor aspects using artificial intelligence, which will immediately appear in the interactive game Aero Gesture. In addition, research is needed to connect this Aero Gesture game with an online system and connect directly to all users, as well as the interconnection between AeroGesture users which at any time can be easily monitored by the user, and the user can easily improve its capabilities with the desired development targets. Future research will also be carried out to obtain a correlation value that is close to significantly associated with the total amount of time achieved.