Sentiment Extraction in Music

Size: px

Start display at page:

Download "Sentiment Extraction in Music"

Brittany Hart
5 years ago
Views:

1 Sentiment Extraction in Music Haruhiro KATAVOSE, Hasakazu HAl and Sei ji NOKUCH Department of Control Engineering Faculty of Engineering Science Osaka University, Toyonaka, Osaka, 560, JAPAN Abstract This paper describes sentiment extraction in music. Sentiments are based on cognition and perception. n this paper sentiments are extracted from real performance through three steps; transcription, musical primitive analysis and music understanding. in the transcription step, acoustic signal is transformed into notes. Musical primitives such as chord progression, melody, rhythm and tempo are analyzed from the notes using musical knowledge. Musical primitives are meaningful for the atmosphere of music. Sentiments are extracted by firing the heuristic rules representing the relation between musical primitives and sentiments for music understanding. - introduction A technic has been used widely in music field(l1. Recently, the question "Can machine possess emotion?" is going to be studied as an application of A. We have been studying machine performance of human-like perception of information in the field of music. t includes recognition, understanding, composition and arrangement. The process simulating human should concatenate the signal processing and knowledge processing. Signal processing handles physical data and knowledge processing treats symbols. We have been constructing the system under the concept shown in Fig. 1. This paper is focussed on sentiment extraction in music. process to extract sentiments is realized in bold lines of Fig. 1. Sentimental information is based on cognition and perception. n music fields, Listening (transcription) corresponds to perception. Musical primitive extraction (analysis) from notes corresponds to cognition. Musical primitive as chord progression or rhythm carries sensitivity information as well as structural information of music. Thus, Melody Rhythm Chord Style Tone color etc Musical Primitives 1 f o A o ) 7 1. \ Fig.1 Concept of System for Music Processing sentiments are extracted if the relation between musical primitives and sentiments are implemented as heuristic rules. 2. Transcription Sentiment extraction has to be based on the analysis of real performance. Therefore transcription, which extracts note (music code) as symbol from acoustic signal, is very important as a means of data input. We have developed transcription system for performed polyphonic music. Po 1 yphon ic transcript ion requires the sound model of instrument (envelope, tone color). The piano is chosen in this paper. Popular light music is selected for the experiment. The transcription system for real performance faces with the following problems. 1. dentification of the note from complicated frequency distribution. 2. Extraction of attack of the note and its duration. CH2614-6/88/0000/1083$ EEE 1083

2 2.1 Human behavior provides us with much information to cope with these problems. The processing concept is shown in Fig.2. This system consists of control module and low level subroutines. Control module, written in OPS83, selects appropriate subroutine with a set of parameters according to processing status. As a remarkable function, this system utilize musical knowledge by the feedback of music analysis as predicting agogics (delicate variation of tempo or rhythm which is added to make the performance expressive). Extraction of Frequency Nap Music code is extracted from the time-frequency map shown in Fig.3; the horizontal axis is time and the vertical is frequency shown by the unit of cent. Pitch and power are extracted from acoustic data by using interpolation method in complex spectrum31. Time-frequency map is obtained by moving observing- w indow. 2.2 Outline of Transcription We can clap hands to the music, which is a noticeable action in listening to the music. t means Control module 1 Symbol extraction and check 2 Control of low level module 3 nstitution of thresholds Processing module Extraction of beat length 2 Search Cor attack post10n 3 Recognition acordlng to volume 4 Recognltlon of the fundamental 5 Recoqnition of the attack 6 Search for contlnuous note 7 Recognitlon of the note shorter than beat A Analyzing module Melody recognition Chord recognition Rhythm recognition OPS83.!Gconstructing routines 1 ' C-ianauaad Fig.2 Concept of Transcription System Fig.3 Time-Frequency Map - c1 --+ Time that we easily recognize the length of the beat and anticipate the instant when the next note will start. Another human-like processing is recognition of the instrument which sounds. t is recognized by the inference of the polyphonic combination of sound model. The processing flow is shown in Fig.4. Preprocessing includes compensation of power, pitch assignment to twelve-tone scale and power smoothing by median filtering. 2.3 Search for beat (decision of attack position) Real performance has agogics and extracted data include disconnection of tone by mis-extraction of peak frequency. n this system, the next optimal attack is extracted using the length of the beat which is adapted by averaging as the music progresses. The primary length of one beat is obtained from the histogram of the duration between the adjacent attack candidates which are extracted by differentiation of powers of the beginning part of music. 2.4 Tone Recognition This section deals with recognition of the tone which is segmented by the above processing. To put it in the concrete 1) The tone is extracted in order of loudness. 2) Whether the tone is fundamental or harmonics is decided by using tone color table. 3) Whether the tone is attacked or continuous is determined by comparison of power before and after beat. 4) The tone whose length is 1 /2 or 1 /4 of one 1084

3 j Compensation of power Pitch assignment to 12-tone scale)... structure pz,c*>. :....: 1 Search for attack position Tone recognition... J4 lmelodyl Estimation of beat length k Symbol processing Fig.4 Processing Flow of Transcription beat has to be recognized. Some explanation is added to 21, which is essential problem of transcription. Every note has the second, the third and higher harmonics, besides the fundamental tone. To cope with this difficulty, table of the harmonics ratio per the fundamental is set in advance. n the first step, every note which may produce the tone under investigation as a harmonic is picked up. The threshold to decide whether the tone under investigation is fundamental or not is obtained from the sum of value given by (Power of picked up note) * (Harmonic ratio to fundamental ). 3. Musical Primitive Extraction The process "How we listen to the music" can't be realized only by recognizing tone sequences. Music has to be interpreted from extracted structures as melody, rhythm, chord, form etc; here we name them "musical primitives". This section deals with extraction of musical primltives from symbols in the structural form as shown in Fig Melody Extraction There are two approaches to extract melody. The first is based on signal processing and the second on musical constraint. n our approach, the following heuristic rules are appiied to extract melody. 1) Melody is the sequence of the loudest tones. 2) Melody consists of the highest note in chord. 3) The pitch difference between adjacent notes is limited [41. These conditions make it possible to extract almost Fig.5 Structure of Husical Primitive all melody. But melody by low tone can't be extracted, however the bass sometimes takes charge in melody. 3.2 Rhythm Pattern Recognition Rhythm is repetition pattern we feel tones as a group. Rhythms are recognized by powers of tones and chord transition. Rhythms which we feel natural are known as two beat, three beat, four beet and so on. We recognize rhythm by top-down searching usiny modulo. This enables us to recognize pieces which include auftact. 3.3 Chord Recognition Chord is very important clue to understand musical atmosphere, especially in western music. t is recognized using the musical knowledge, which is divided into the following two big categories. 1. Knowledge which assign chord-name to the local set of tones. 2. Knowledge which merge the local chord-names. Fig.6 shows the outline of chord recognition step. Arpeggio is also recognized here. 3.4 Key Recognition Common approach to recognize the principle key utilizing histogram of the number of used tones has to face with modulation problem and confusion by 1085

4 progressing notes. t is caused by not taking time transition into consideration. n this paper, chord progression and tensions are utilized. This approach makes it possible to decide the principle key with a few data and to detect easily when the key changes. - Melody Uoth of the fourth and the seventh don't exist. Orienlal mood Transcription W.M. Music code 83 G'3 C4G3 (G'3) D4(C4)F'3. Rules to assign local chord name 2. Rules to merge chord names Chord Extraction by Musical Rules Cliord progression Dm-+ G -+CmaJ7+ F-+ Om7 +E7-+Am Plaritive mood Key = )c- lurol mood Khylhm 0 beol and fost t Cheerful mood Extracted chor Local hsslgnrnent 5-8 Dmin7 Search for root tone based on root tone 1 - Conflict resolution Search for the range where the chord holds : Global Assignme/nt Flow in time area Fig.6 Outline of Chord Extraction 4. Husic Understanding We define that music understanding is to have sentiment for music. For its realization in the machine, it is available to enumerate heuristic rules of relations between extracted musical primitives and sentiments. The rules used in this system are generated from those who are experienced in music through the interview process. Each rule may be a trivial one. But they would produce respectable complicated sentiments, supposing that music primitives are related with more densely in the form as Fig.5. Examples of rules used here are shown in Fig.7. n addition to these rules, the rules which summarize and output obtained sentiments are registered. Fortunately, in the music field, not a few psychological experiment are reported. These results suggest the heuristic rules to be implemented. Fig.7 Example of Rules to Extract Sentiments 5. Experiment Sentiments are ext.racted from acoustic signal by way of music code and musical primitive. Each experimental result is shown in this paragraph. The beginning part of extracted music codes of piano music called "Light and shadow of the youth is shown in Fig.8. Experimental results for some pieces are shown in Table 1. Recognition rate is defined as below. Recognition rate = (number of extracted right notes - extra extracted wrong notes) / (total number of notes) Fig.9 shows the process to extract musical primitives and sentiments from the data shown in Fig.8. The sentiments extracted from the whole music "Light and shadow of the youth" are shown in Fig. 10. This result shows that whole atmosphere of this music is sorrowful and hopeful impression is observed on the release. This estimation seems to correspond with the title of the music. n this paper musical primitive is used to extract sentiments. This approach has the problem how much musical primitives can be extracted. The style of music or playing characteristic as agogics have to be recognized as well as music primitive described above. Another difficulty is left in the definition of relation between musical primitive and sentiments. This is the definition problem of individual appreciation model, while necessary rules are extracted through interview process from the specified persons this time. SD-method of psychological fields can be utilized to make the general appreciation model. 1086

5 6. Conclusion We have described sentiment extraction through three steps in music information processing. The performance is insufficient compared with human. But it deserves our attention that the process "listening to the music" is executed by the machine. Each process leaves the room for exploration. This fact, from an another view, ensures the possibility that machine draws nearer the human. This paper payed attention to technological aspect to extract sentiment in music. But the field that deals with "Cognition" should be interdisciplinary in itself. For the completion of the system, closer connection with musicology or psychology is indispensable. Format countlng no. tonol tone2... attacked tone tone-nams. vo\oslty G G 7 U 9 O d G G speed contlnuous tone (tone-namo) lbeat - 0.4G second Fig.8 Extracted Music code Table 1. Re s U 1 r - 7 Recosn t on 1 CYninoPedlel by Satle Setsvun ti0 lllcarl To Kage by Ken Muraniatsu KacrnicJi by Ken Mur~arrralsu References 111 C. Roads : Research in Music and Artificial ntelligence, ACM Computing Surveys, Vol. 17 (1985) 12) C. Chafe and D. Jaffe : Source Separation and Note dentification in Polyphonic Music, CASSP86 ( 1986) 131 M. lmei and S. lnokuchi : Frequency dentification by Complex spectrum, CASSP86 < 1986) 41 A. J. Watkins : Perceptual aspects of synthesized approximations to melody, J. Acoust. Soc. Am. 78 ( 1985) Fig.9 <.Musical Premitives.ount b ( 1 12 S 14 5 el 17 re melody /', A3 63 CJ U4 E4 \.. D4 chord tension' LG,oomy / / i i l... j l... Process to Extract Sentiment from the data of Fig.8 Whole atmosphere of this music is slow and leisurely. sorrowful. smart and serious. Level is 10. urbane. Level is 7. rural. Level is 6. There is pathetic mood on chord from 5 to 33. There is pathetic mood on chord from 37 to 65. There is hopeful mood on shord from 69 to 97. There is melancholy mood on chord from 105 to 116. There is pathetic mood on chord from 133 to 153. On release at 75, serious mood. there is hopeful, urbane, smart, Fig. 10 Sentiments Extracted from Piano Husic 'Light and Shadow of the Youth' 1087

TOWARD AN INTELLIGENT EDITOR FOR JAZZ MUSIC

TOWARD AN INTELLIGENT EDITOR FOR JAZZ MUSIC G.TZANETAKIS, N.HU, AND R.B. DANNENBERG Computer Science Department, Carnegie Mellon University 5000 Forbes Avenue, Pittsburgh, PA 15213, USA E-mail: gtzan@cs.cmu.edu