A System for Memorizing Chinese Characters using a Song based on Strokes and Structures of the Character

A System for Memorizing Chinese Characters using a Song based on Strokes and Structures of the Character Yuma Ito Grad. Sch. of Engineering, Kobe University 1-1 Rokkodaicho Nadaku, Kobe, 657-8501, JAPAN yuma-ito@stu.kobeu.ac.jp Tsutomu Terada Grad. Sch. of Engineering, Kobe University, PRESTO JST 1-1 Rokkodaicho Nadaku, Kobe, 657-8501, JAPAN tsutomu@eedept.kobeu.ac.jp Masahiko Tsukamoto Grad. Sch. of Engineering, Kobe University 1-1 Rokkodaicho Nadaku, Kobe, 657-8501, JAPAN tuka@kobe-u.ac.jp ABSTRACT Hundreds of thousands of Chinese characters are used in the Japanese and Chinese writing systems. Memorizing all the Chinese characters requires great effort because learners have to memorize not only the shape of the character but also its meaning and usage. Research to find a method to present the shape of the figures using sound has been conducted. Consequently, it is possible to generate a song for learning Chinese characters comprehensively using sounds that represent the shapes of the characters and lyrics that represent the meaning and usage. We propose a method to generate a song based on the structures, s, and usage of Chinese characters and a method to memorize Chinese characters using the generated song. Categories and Subject Descriptors K.3.1 [Computers and Education]: Computer Users in Education Computer-assisted instruction (CAI) Keywords Memorizing Chinese characters, Generate song, Learning support 1. INTRODUCTION It is important to learn the characters when learning languages. In particular, the Japanese and Chinese writing systems have the most characters of all. The number of characters is far more than a hundred thousand. Even just considering the Chinese characters defined by Unicode [1], which is one of the character codes, the number of characters is approximately 70,000. Learning all the Chinese characters is difficult because it requires memorizing not only the shape of the characters but also the orders and meanings. Although various systems and researches have been developed for learning the shapes and s of a single Chinese Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. iiwas 15, December 10-12, 2015, Brussels, Belgium c 2015 ACM. ISBN 978-1-4503-3491-4/15/12... $15.00 DOI: http://dx.doi.org/10.1145/2837185.2837235 character [2, 3], there is no method for learning the meanings, usage examples, and idioms of Chinese characters at the same time. On the other hand, Oxford et al. [4] argues that the efficient way to learn the vocabulary of a foreign language is correlating the words with a sense such as visual sense, auditory sense, and tactile sense. Moreover, it is important to read aloud the meaning and an example sentence that the word is actually used in. Abboud et al. [5] researched a system that presents the shape and color of the basic figure and simple characters to a person using sound. In this way, it is possible to construct a system for learning Chinese characters using sounds and other auditory stimuli. Therefore, we propose a system for memorizing Chinese characters using a song based on s and structures of the character. According to Yiu et al. [6], there are 36 kinds of s. We allocate each to a melody and describe the shape of the Chinese character by the melodies. In addition, a Chinese character has a fixed structure, such as a left component and right component. We define the component part of the Chinese character as component and the combination of components as structure. Figure 1 shows examples of components and structures of Chinese characters. There is a Chinese character in each square. The red area and blue area show the components of the Chinese character in each square. Moreover, the name of the component is written under the character. Because of these structures and components, we can recognize the rough shape of Chinese characters. We allocate the structure and components to a chord that represents the mood of the song. Accordingly, the sequence of the sound generated in accordance with the s becomes a melody that follows musical theory. This enables learners to perceive and memorize the melody easily. Finally, we allocate the meaning and example sentence of the Chinese character to the lyrics of the melody. Learners can learn the shape of a Chinese character by melody and chord progression and the meaning and the usage by the lyrics. 2. RELATED WORK There are several researches relating to supporting learning Chinese characters [2, 3]. These researches propose a method for learning the shape and s of Chinese characters for elementary school students and non-native Chinese speakers who have never learned Chinese characters. Teo et al. [2] proposed a system to learn the shape and s of

Left component Right component (a) Surround from lower left component Surrounded component (b) Input Chinese characters Proposed system Database of Chinese character information Composed of * Stroke data * Component data Generate chord & melody Figure 1: Examples of component and structure Learner Database of example sentences Chinese characters by haptic interface. Additionally, Wen et al. [3] argued the system needs to learn the structure of Chinese characters. They focus on the s and structures of Chinese characters. Thereby, we regard the s and structures as an important element. However, Chinese characters are generally used in idioms, which are composed of two or more Chinese characters and have a special meaning. Learners cannot learn the idioms of Chinese characters by these methods, because they focus on learning one character at a time. Furthermore, because there are many homonyms, where the pronunciation is the same but the meaning is different, it is necessary to learn the difference between them. Nevertheless, these methods do not support the learning of homonyms. Accordingly, we constructed a system where learners can learn all of the s, orders, meanings, and usage examples for Chinese characters including idioms. Abboud et al. [5] proposed a system that presents the shape and color of a figure and simple characters using sound. This system allocates a note of different pitch to each pixel in a vertical direction and scans the figure from the left. It represents the shape of the figure by sounding notes corresponding to the pixels on the scanning line. Abboud et al. state that the user can recognize the basic figure, such as a rectangle, triangle, and some lines. Although people can recognize a figure by sound as mentioned in this research, this method is not appropriate for Chinese characters due to their complexity of shape. Thus, we propose a new method to present the shape of Chinese characters. 3. SYSTEM DESIGN Methods of presenting characters such as Chinese characters include the following: Presenting absolute position This method includes Abboud s method [5] mentioned in section 2 and the method that allocates the sounds to each coordinate of an image of a character. However, it is difficult to memorize all the sounds that are allocated to each coordinate because of the complexity of shape. Presenting direction A Chinese character can be described as an assembly of s. Therefore, this method includes the method that presents direction. Presenting structure Chinese characters have certain structures as mentioned in section 1. Because these structures give the rough positions of s and meanings, it is appropriate to present the structures to learn Chinese characters. Output song Composed of example sentences Allocate lyrics Figure 2: Processing flow of proposed system In our research, we apply a method of presenting structure to our proposed system because we focus on learning not only the shapes of Chinese characters but also the meanings and usage examples. 3.1 Outline of proposed system Figure 2 illustrates the processing flow of the proposed system. The proposed system has two databases; one is composed of data and component data of Chinese characters, and the other is composed of example sentences. As shown in Figure 2, the proposed system searches for the component data and data for each Chinese character input by the learner. The system generates a chord and melody based on the component data and the data. Moreover, after searching for a suitable sentence in the example sentence database, the system allocates it to the generated melody. Finally, the proposed system outputs the song. Next, we explain the usage example. Learners firstly recall the song that is used at learning time when he/she wants to recall the Chinese character when learning or taking an examination. At the learning time, such as when preparing for an examination, learners train to recall the melodies and Chinese characters by listening to the song. In contrast, at the time of actual practice, such as taking an examination, learners recall the character information without listening to the song. After recalling the melody, learners write down Chinese characters by recalling the shape of the characters. In our research, the proposed system outputs the song in accordance with the following three phases: converting components to chords, converting s to melodies, and allocating lyrics. We explain the phases in the following sections. 3.2 Converting components to chords As mentioned above, the structure gives the rough shape and meaning. Therefore, we allocate the structures to the chord progression that describes the mood of the song. Unicode [1], which is a character code, defines 12 kinds of ideographic description sequences (IDS). By using IDS, it is possible to describe 13 kinds of structures including structures that cannot be divided any further, such as and. In accordance with IDS, because there are at most three

Table 1: Allocation between structures and chord progressions Structure Chinese character Chord progression Single, C (a) Left to right, C, G (b) Above to below, Am, G (d) Left to middle and right, C, F, G (c) Above to middle and below, Am, G, E (e) Fully surrounded, Bm, A (d) Surrounded from above, Cm, B (d) Surrounded from below Dm, C (d) Surrounded from left, D, A (b) Surrounded from upper left, Em, D (d) Surrounded from upper right, E, B (b) Surrounded from lower left, F, C (b) Overlaid, C, Cm (f) components that compose one Chinese character, the chord progressions should be composed of one to three chords. We propose 6 kinds of chord progressions that are often used in popular music and expand them into 13 kinds of chord progressions by changing the root note. Firstly, we present the 6 kinds of basic chord progressions in the following list. Note that the chords in the following list are indicated by Roman numerals, and the descriptions in the brackets are the examples when the root note is C. (a) I (C) (b) I, V (C, G) (c) I, IV, V (C, F, G) (d) Im, VII (Cm, B ) (e) Im, VII, V (Cm, B, G) (f) I, Im (C, Cm) Table 1 shows the allocation between structures and chord progressions that are actually used in this research. The left column shows the name of the structure and its figure, the middle column shows the corresponding Chinese characters, and the right column shows the chord progression and the symbol of the chord progression mentioned above that underlies the chord progression in this table. Furthermore, each chord of the chord progressions corresponds to each component shown in white, gray, and black. Note that for the overlaid structure, we allocate the C chord to the component written at first on the order and the C minor chord to that written at second. 3.3 Converting s to melodies According to Yiu et al. [6], Chinese character s are classified into 36 kinds of s. Moreover, they can be subdivided into 10 subs. Figure 3 shows examples of components, s, and subs of a Chinese character. From the top rows describe Chinese character, components, s, and subs. The corresponding component is colored red in the components row in the figure. Additionally, the names of the subs are described in the subs row. As shown in Figure 3, the Chinese character is classified as having the structure of surrounded from upper right. The component of upper right has two s, and that of lower left has three s. In other words, this Chinese character has five s. In addition, these five s are subdivided into eight subs. In our research, we allocate the subs to a one-beat melody. The proposed system firstly divides all the s belonging to the Chinese character into subs. After that, the system converts the subs into melodies and combines the melodies into a song. The reason the length of the melody generated from the subs is one beat is that it is easy for a person to recognize a melody of this length. A melody following musical theory enables learners to recognize and memorize it easily. We use Orpheus [7], which is an automatic composition system, to generate melodies. Orpheus generates melodies from chord progression, rhythm, and pitch transition probability defined by several parameters such as lyrics and tune using dynamic programming [8]. In our research, we define rhythm and pitch transition probability from subs. Additionally, the proposed system generates melodies using dynamic programming in the same

Chinese character Components Strokes Subs Falling leftward Rightward Downward Change in direction Downward Rightward Downward Rightward Figure 3: Example of components, s, and subs way as Orpheus. The melody generation method has three sections: definition of pitch transition probability, allocation between subs and transition probabilities, and connection of the melodies. 3.3.1 Definition of pitch transition probability Figure 4 shows the allocation between subs and melodies. The subs and their names are described on the left, and melody patterns are presented on the right. The horizontal lines in the figure represent the length of one beat. Furthermore, vertical lines are drawn at every quarter beat. The red arrows in Figure 4 represent the degree of pitch transition. The proposed system calculates the pitch transition probability based on this degree of pitch transition. We propose five kinds of degree of pitch transition: stable, falling, rapidly falling, rising, and free. Note that four of them are illustrated in Figure 4. We define the number of notes belonging to a melody as N and the pitch of the i-th note as the MIDI note number n i (1 i N), where the range of pitch is from C3 to C5 (60 n i 84). At this time, we denote the pitch transition probability between n i and n i+1 by p(n i, n i+1 ). The pitch transition probability p(n i, n i+1 ) in each degree of pitch transition are defined by Equation (1) (5). In the following equations, n and c(n i+1 ) are described by the following two equations. n = n i+1 n i, 1 n i+1 belongs to the chord, c(n i+1) = 0.2 n i+1 does not belong to the chord. Stable As shown in Figure 4, because there is no transition in pitch, p(n i, n i+1 ) is described by Equation (1). 1 n = 0, p(n i, n i+1) = (1) 0 otherwise. Falling The pitch falls by at most five semitones as shown in Figure 4. Therefore p(n i, n i+1) is denoted by Equation (2). c(n i+1 ) 5 n 0, p(n i, n i+1 ) = (2) 0 otherwise. Rapidly falling As shown in Figure 4, the pitch falls by more than the parameter of falling. p(n i, n i+1 ) is de- Rightward Downward Falling rightward Falling leftward Right curve (Horizontal) Right curve (Vertical) Left curve Tiny dash Change in direction Flick up and rightward Stable No transition in pitch Falling Falling at most 5 semitones Rapidly falling Falling at least 5 semitones Rising Rising at most 5 semitones Figure 4: Allocation between subs and melodies fined by Equation (3). c(n i+1) n < 5, p(n i, n i+1 ) = n /10 5 n 0, 0 otherwise. Rising Because the pitch rises slightly as shown in Figure 4, p(n i, n i+1 ) is described by Equation (4) c(n i+1 ) 0 n 5, p(n i, n i+1) = (4) 0 otherwise. Free This parameter allows a note to jump to a nearby pitch. Therefore, p(n i, n i+1 ) is denoted by Equation (5). c(n i+1 ) n 5, p(n i, n i+1 ) = (5) c(n i+1)/4 otherwise. (3)

3.3.2 Allocation between subs and transition probabilities We explain the allocations between subs and melodies in Figure 4 and the following items in this section. Note that the titles of the following items correspond to the names of the subs presented in Figure 4. Rightward This is represented by one quarter note because its line does not move in a vertical direction. Downward Because the line of this falls vertically, it is denoted by two eighth notes with rapidly falling. Falling rightward This is used, for example, for the second of and the third of. Therefore, we describe it as four sixteenth notes connected to falling. Falling leftward This is represented by four sixteenth notes as well as falling rightward. However, the degrees of pitch transition of each note consist of stable, falling, and stable. Right curve (horizontal) This is used only at the second of and the third of. Additionally, change in direction always follows it. This is described by two sixteenth notes and one eighth note connected to falling and stable. Right curve (vertical) This is used, for example, for the fifth of. Change in direction as well as right curve (horizontal) always follow it. We describe it by the sequence of notes that is composed of two sixteenth notes and one eighth note connected to rapidly falling and stable to express the direction as more vertical than that of right curve (horizontal). Left curve This is used, for example, for the sixth of. Moreover, change in direction always follows it. This is represented by one eighth note and two sixteenth notes connected to stable and falling. Tiny dash Tiny dash in both the right direction and left direction are denoted as tiny dash. For example, the first to the fourth of correspond to tiny dash. The is described by one eighth note and one eighth rest to express this short. Change in direction This is not an independent and always follows another. For example, this is used at a such as the end of the fifth of and the end of the second of. Although this is a short like tiny dash, this is connected to another with rising. Flick up and rightward This is used, for example, for the fifth of and the fifth of. This, like tiny dash and change in direction, is often short. Therefore, a rest is put at the latter half of the beat and two sixteenth notes connected to rising are put at the first half of the beat. Chinese character Component Stroke Stroke Stroke Stable, Rising No rest Free No rest Component Chinese character Component Stroke 1 beat rest 2 or more beat rest Figure 5: Connection method 3.3.3 Connection of melodies The proposed system successively generates a note from the first sub based on the method mentioned in the previous sections. In this section, we explain the connection method between the note at the end of a and the note at the start of the following. Figure 5 shows the connection method. The sequences of notes in Figure 5 represent the rhythm of a melody generated from subs. The rectangles labeled, component, and Chinese character surround the melodies generated from subs that are composed of each, component, and Chinese character. Additionally, the connection method is shown at the bottom of the figure. The proposed system connects the melodies by stable or rising (if the following sub is change in direction) if the two melodies belong to the same and connects by free if the melodies belong to a different. Moreover, the system does not put a rest in these connection cases. Next, when the two melodies belong to a different component, the system puts a 1 beat rest between the melodies. This enables the melody to have a block and learners to memorize Chinese characters by memorizing every component. Finally, the system puts a 2 or more beat rest between the melodies belonging to the different Chinese characters. This prevents learners from confusing this connection with the connection between the components. Note that the proposed system calculates the appropriate length of the rest so that the first note of the Chinese character always begins at the first beat of the measure. This enables learners to recognize the beginning of Chinese characters easily. Furthermore, the notes circled in red in Figure 5 represent the first notes of the components. The system allocates them to the root note of the chord that is allocated to the component. 3.4 Allocation of lyrics Because the lyrics are the most important key for learners to recall the melody, it is necessary to have lyrics that can be recalled easily. Thus, we propose a method to use an example sentence where the Chinese character is actually used in the lyrics. Let Y be a text string of an example sentence including a Chinese character or an idiom, and X be a text string of the pronunciation of the Chinese character or one Chinese character belonging to the idiom. The system connects Y and X by the Japanese postpositional particle no, which corresponds to the preposition of in English, and let the connected text string be the lyrics. Note that if there are not enough notes to allocate the example sentence, only the pronunciation will be represented in the lyrics. In addition, we propose a method to allocate the lyrics to the melody equally. Figure 6 shows an example of equal

Chinese charcters: Example sentences: for for /fu-ka n/ overlook /ta ka da i ni no bo t te shi ga i wo fu ka n su ru/ overlooking the town from the hill /ta ka da i ni no bo t te shi ga i wo fu ka n su ru/ overlooking the town from the hill Figure 7: Example of generated melody Figure 6: An example of the equal allocation where k t = arg max M k, (6) 1 k K M k N allocation. The sequence of the notes at the top of the figure represents the generated melody, and the squares at the bottom of the figure represent the mora of the lyrics. The mora is a unit in phonology, which is often used in Japanese. Furthermore, most moras are composed of only one consonant and only one vowel in Japanese. For example, the word cho-ko-re-e-to, which means chocolate, has five moras. In the example shown in Figure 6, the lyrics of four mora are allocated to eight notes of a melody. For the notes that have no allocation, the system allocates the mora vowel of the note just before. Accordingly, it is important to reduce the number of notes that have no allocation to create a song that is easy to listen to. Because the number of notes depends on the number of s of the Chinese character, the system generate lyrics for each Chinese character. We define Y k (1 k K) as all example sentences with regard to a Chinese character or an idiom, X as the text string of pronunciation of one Chinese character, m(x) as the number of mora for the text string X, and N as the number of notes that compose the melody. The lyrics are provided by Equation (6). M k = m(y k ) + m(x) + 1. If the system cannot find an example sentence that satisfies Equation (6) due to lack of notes, the system uses X itself, which is the pronunciation of the Chinese character, as the lyrics. 3.5 Example of generated melody We present an example of the generated melody. Figure 7 shows the musical score of the melodies *1. Firstly, the table at the top of Figure 7 shows the information of Chinese characters and example sentences. The text strings enclosed by slashes represent the actual pronunciation in Japanese. Additionally, white space and hyphens are inserted between moras and Chinese characters, respectively. The underlines show the text that corresponds to the Chinese character. Secondly, the lower part of the figure shows the musical score of the melodies that are generated by the proposed system. The two rows above the score denote components, which are colored red, and s from the top. The s and the *1 The sound of the example is uploaded here. https://youtu.be/qmhsnsykfsc

overlooking the town from the hill /fu-ka n/ overlook nettle rash spread over the body /ji n-ma-shi n/ nettle rash Figure 8: Screenshot of prototype system rectangles that surround the s are positioned above the corresponding melodies. In the example of Figure 7, the melody is generated from the idiom of ; the pronunciation of which is /fu-kan/, and the meaning of which is overlook. Since both of the structures of each Chinese character are left to right, the proposed system allocates the chords C and G to the chord progression. The number of notes of each Chinese character are 25 and 32. Accordingly, both of the example sentences chosen by the system are /ta ka da i ni no bo t te shi ga i wo fu ka n su ru/ in the pronunciation description, which literally means overlooking the town from the hill. As shown in the lyrics of Figure 7, the final lyrics are the text strings of the example sentences (/ta ka da i ni no bo t te shi ga i wo fu ka n su ru/) and the pronunciations of Chinese characters (/fu/ and /ka n/) connected by the postpositional particle no. 4. IMPLEMENTATION We implemented the prototype system mentioned in section 3. Figure 8 shows the screenshot of the prototype system. The prototype system has text boxes for inputting a Chinese character and its pronunciation at the upper left of the figure and for inputting an example sentence at the upper right of the figure. When the button on the upper right of Figure 8 is pushed, the prototype system outputs the lyrics and melody in the format of musicxml [9]. Simultaneously, the system presents the Chinese character, its structure, and its s as shown in Figure 8. The prototype system has a dataset of Chinese characters, their structures, and their s in XML format and search data from input Chinese characters. We used the data of 20,903 Chinese characters from the CHISE [10] (CHaracter Information Service Environment) as the structure data. With regard to data, we input data of 2,136 Chinese characters manually. We developed the prototype system using Microsoft Visual C# 2013. 5. EVALUATION In our research, we evaluated the proposed system using the prototype system. 5.1 Evaluation procedure We evaluated the proposed method compared to the method of memorizing the Chinese characters without any learning Figure 9: Example of question sheet support. The participants memorized 15 idioms of Chinese characters in both methods. We measured the number of recalled Chinese characters after three days. The details of each method are as follows: Conventional method Memorizing Chinese characters by only looking at a list that consists of Chinese characters and their example sentences. Proposed method Memorizing Chinese characters by listening to the generated sound and using the list that is the same as that used in the conventional method. Note that the participants were permitted to speak and move their mouths, but not permitted to write down the characters during the experiment. The evaluation procedure is as follows: (i) The participants received a question sheet and answered the dictation questions of Chinese characters. We call the answers answered at this time before learning. (ii) The participants memorized the Chinese characters using each method until they had memorized them completely. (iii) Three days after finishing memorizing the characters, the participants answered the same dictation questions as used in phase (i). We call the answers answered at this time after learning. Additionally, the participants who used the proposed method were permitted to listen to the song used in phase (ii) if they are completely unable to recall the answer. They were required to write down whether they had to listen to the song to answer the question. Figure 9 shows an example of the question sheet used in phase (i) and phase (iii). The question sheet has example sentences. Note that the question sheet that is actually used in the evaluation is written only in Japanese. There are no annotations like the blue and orange rectangles shown in Figure 9. Chinese characters for sentence questions are shown with other Japanese characters (called katakana), which are underlined, that represent the pronunciation. The text in the blue rectangle in Figure 9 is the pronunciation text of the Chinese character for a question using the alphabet and

the text that is translated into English. Hyphens and white spaces written in the pronunciation text are inserted between Chinese characters and moras, respectively. In other words, the example at the top is three moras and is described by two Chinese characters, and the example at the bottom is five moras and is described by three Chinese characters. Moreover, the text in the orange rectangle in the figure is the text of the example sentence translated into English. The participants are required to answer 15 questions in five minutes. Note that the order of the example sentences for questions used in phase (i) and phase (iii) is different. In addition, the sentences used in phase (ii) are the same as those written on the question sheet. 5.2 Participants There were six participants, all of whom are native speakers of Japanese. Furthermore, they are all graduate school students who major in information engineering. Three of them memorized the characters by the conventional method, and the others memorized them by the proposed method. 5.3 Questions and songs We took the 15 dictation questions for this evaluation from level 1 of the Japan Kanji Aptitude Test held by the Japan Kanji Aptitude Testing Public Interest Foundation [11]. The reason we chose level 1 is that it is difficult for a native speaker of Japanese to guess the correct answer. Note that the 15 questions we chose for evaluation include 25 Chinese characters. The song used in this evaluation is generated as follows: Convert the musicxml file generated by the prototype system into a voice file (WAV format) using Ce- VIO [12], which is voice synthesis software. Generate the song file (MP3 format) by giving the piano accompaniment as chord information to the voice file using GarageBand [13], which is software for creating music. 5.4 Result and discussion Table 2 shows the number of Chinese characters that the participant answered correct. The values outside the brackets are the sum of the correct answers without listening to the song, and the values in the brackets are the sum of all the correct answers. Additionally, the values on the bottom row represent the mean and the standard deviation in each method. With regard to the correct answers without listening, according to the result of analysis of variance, there is no significant difference between the methods in before learning (F (1,4) = 0.08), and there is a significant tendency between methods in after learning (F (1,4) = 6.32, p <.10). Similarly, with regard to the sum of all correct answers, according to the result of analysis of variance, there is no significant difference between methods in before learning (F (1,4) = 0.08), and there is a significant tendency between methods in after learning (F (1,4) = 6.04, p <.10). As a result, there is no advantage when using the proposed method. We discuss what causes this result. Firstly, we discuss whether the participants were able to associate the melodies with s. All the participants who memorized the characters by the proposed method stated that although they could have associated the melodies, they could not recall the melodies and the characters in the after learning. It is Table 2: Number of correctly answered Chinese characters Method Participant Before learning After learning P1 2 19 P2 2 24 Conventional method P3 2 23 Mean 2 22 S.D. 0.00 2.16 P4 4 16(17) P5 3 17(17) Proposed method P6 0 6(7) Mean 2.3 12.7(13.3) S.D. 1.70 4.78(4.50) assumed that this was caused by the shortness of the memorizing time and being unfamiliar with the melodies generated by the proposed system. Accordingly, we should improve the evaluation method by, for example, setting the training term so that the participants are familiar with the melodies of the proposed method. Secondly, there is the opinion that it is necessary to connect visual information and auditory information. Therefore, we should propose a visual presentation method such as highlighting the corresponding to the sound that is currently played. Finally, a participant stated that while he was not aware of the lyrics just after beginning memorizing, he become aware of them gradually. Furthermore, another participant mentioned that because it is good for learning Chinese characters to recall them by an example sentence, it is necessary to use example sentences as lyrics. This supports the necessity of using example sentences as lyrics. 6. APPLICATION EXAMPLES Finally, we explain the application examples of the proposed method. The main feature of our research is representing information, such as the s of Chinese characters, and enabling learners to associate the s and the meanings of the characters with music. There are several situations when we have to memorize information without memorizing characters. We explain the application examples with Figure 10 and the following comments. Memorization of ideographic commands and motions There is some application software that requires motion, such as Graffiti of Palm [14] and Glyph of Ingress [15]. Figure 10(a) shows an example of ideographic command. Another examples is memorizing the motion of a magic wand as shown in Figure 10(b). This motion often appears in fantasy stories, movies, and video games. To make the game more enjoyable, it is important to memorize these motions. Memorization of route Figure 10(c) shows an example of the route within a building. The proposed method can be applied to memorize a short cut in a building or a route from the exit to the parking location as shown in the figure. 7. CONCLUSION AND FUTURE WORK We proposed a system for memorizing Chinese characters using a song based on the s and structures of the characters. The proposed system supports memorizing Chinese

Command A (a) Ideographic command You are here Magic of xxx (b) Motion of magic wand Destination (c) Memorization of route Figure 10: Examples of application characters by the song, which is composed of a melody generated based on the structures and s, and the lyrics using the example sentence. We compared the proposed method with the conventional method. As a result, although there were no advantages when using the proposed method, we obtained many useful ideas for improvements. Future work includes creating a visual presentation method, such as highlighting the corresponding to the sound. We should also improve the experiment procedure by, for example, setting the training term so that the participants can be familiar with the generated melody. Additionally, we intend to develop an appropriate allocation method for lyrics and a melody generation method to make memorizing characters easier. [4] R. Oxford and D. Crookall: Vocabulary learning: A critical analysis of techniques, TESL Canada Journal, Vol. 7, No. 2, pp. 09 30 (1990). [5] S. Abboud, S. Hanassy, S. Levy-Tzedek, S. Maidenbaum, and A. Amedi: EyeMusic: Introducing a Visual Colorful Experience for the Blind Using Auditory Sensory Substitution, Restorative Neurology and Neuroscience, Vol. 32, No. 2, pp. 247 257 (2014). [6] C. L. Yiu and W. Wong: Chinese character synthesis using METAPOST, Proc. of TUG, pp. 85 93 (2003). [7] S. Fukayama, D. Saito, and S. Sagayama: Assistance for Novice Users on Creating Songs from Japanese Lyrics, The 2012 International Computer Music Conference (ICMC2012), pp. 441 446 (2012). [8] R. E. Bellman: Dynamic Programming, Princeton University Press (1957). [9] MusicXML, http://www.musicxml.com/. [10] CHISE: Character Information Service Environment, http://kanji.zinbun.kyoto-u.ac.jp/projects/chise/ids/. [11] Japan Kanji Aptitude Testing Public Interest Foundation, http://www.kanken.or.jp/. [12] CEVIO Official Site, http://cevio.jp/. [13] GarageBand, http://www.apple.com/jp/mac/garageband/. [14] A. Butter and D. Pogue: Piloting Palm: The inside story of Palm, Handspring, and the birth of the billion-dollar handheld industry, John Wiley & Sons (2002). [15] Ingress, https://www.ingress.com/intel. 8. ACKNOWLEDGMENTS This research was supported in part by a Grant-in-Aid for Scientific Research (A)(23240010) and by Super Cluster Program (Kyoto Area, Construction of Highly-Efficient Energy to Achieve a Clean and Low Environmental Impact Society) from the Japan Science and Technology Agency. 9. REFERENCES [1] The Unicode Consortium, The Unicode Standard, http://www.unicode.org/versions/latest/. [2] C. Teo, E. Burdet, and H. Lim: A robotic teacher of Chinese handwriting, Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2002. HAPTICS 2002. Proceedings. 10th Symposium on, pp. 335 341 (2002). [3] Y. Wen, L. P. Prieto, and P. Dillenbourg: Paper-Based Tabletop Application for Collaborative Chinese Character Learning, Proc. of the 11th International Conference on Computer Supported Collaborative Learning (CSCL2015), pp. 665 666 (2015).