AI-based Music Discovery Application Design

Transcription

1 AI-based Music Discovery Application Design MSc Product Service System Design 2017/2018 Double Degree Program Candidate Jan Dornig Scuola del Design Tutor Davide Fassi College of Design and Innovation Tutor Sun Xiaohua

2

3 ABSTRACT With advances made in the field of Machine Learning and increasing consumption of digital content, individualisation of digital services are reaching new heights. As the data those algorithms are trained on, is created by human behavior, the algorithms can be seen as a digital mirror of human actions. Currently the control of most of these algorithms and the data they collect is often innaccessible to users. In this work, we explore the value of using artificial intelligence technology in music discovery and the benefit of giving the user agency over the technology and data. The study examines current music services and relevant literature before reviewing the technological possibilites, necessitites and implications of using machine learning. After conducting user interviews for a human centered design of the service, the work results in the proposal of a music discovery service with an interactive machine learning system in form of a digital application which uses an interactive knowledge graph for visualization and interactive intelligent agents for exploration. Keys words: Music Discovery, Artificial Intelligence, Service System, Machine Learning, User Experience I

4 AI-based Music Discovery Application Design ITALIAN ABSTRACT Con i progressi fatti nel campo dell' apprendimento della macchina e l'aumento del consumo di contenuti digitali, l'individualizzazione di servizi digitali sta raggiungendo nuove altezze. Visto che i data su cui questi algoritmi sono prodotti sono basati su abitudini umane, possiamo vederli come degli specchi delle azioni umane. Attualmente il controllo della maggior parte di questi algoritmi e dei dati che raccolgono è spesso inaccessibile agli utenti. In questo lavoro scopriremo l'importanza di usare la tecnologia di intelligenza artificiale nella scoperta musicale e i benefici di dare all'utente potere oltre che la tecnologia e i dati. Questo studio esamina gli attuali servizi riguardanti la musica e la letteratura attinente, Riesaminando le possibilità tecnologiche, le necessità e le implicazioni di usare l'apprendimento della macchina. Dopo aver condotto interviste per settare un servizio human centered, lo studio porta alla proposta di un servizio di scoperts musicale con un sistema di apprendimento della macchina interattivo nella forma di applicazione digitale, che usa un grafico di conoscenza interattiva per quanto riguarda l'estetica, e agenti interattivi intelligenti per l'esplorazione Keys words: Scoperta della musica, intelligenza artificiale, programma di servizio, apprendimento macchina, esperienza dell'utente II

5 Contents 1 INTRODUCTION Research Background Research Methodology Research Usefulness Scope Music Discovery MUSIC DISCOVERY TOOLS SURVEY Service Benchmarks Heuristic User Interface Evaluation Spotify Pandora Last.fm Allmusic Summary FUNCTIONS OF MUSIC Music and Identity Self-Reflection Music as Therapy Summary THE INFLUENCE OF TECHNOLOGY ON MUSIC CONSUMPTION Describing Music Music Information and Big Data HUMAN AUGMENTATION AND AI IN MUSIC DISCOVERY Interactive Machine learning Agentive Technology Current State Collaborative AI Knowledge Graphs for Exploration III

6 AI-based Music Discovery Application Design Machine Learning Algorithms for Recommendation DESIGN OF A COLLABORATIVE AI- HUMAN MUSIC DISCOVERY SERVICE User Interviews Questions and Answers User Interview Summary Service Positioning User Definition System Architecture Customer Journey Interface Design Validation Interviews CONCLUSION & FUTURE RESEARCH IV

7 List of Figures Figure 1 Spotify UI Browse Figure 2 Spotify UI Artist Figure 3 Spotify Mobile App UI, Home screen (left) Artist screen (right) Figure 4 Spotify General Music Controls (Highlighted) Figure 5 Spotify Discover Weekly Feedback Functionality (Highlighted) Figure 6 Spotify Radio Controls and Feedback (highlighted) Figure 7 Pandora Web UI Figure 8 Pandora Music Style Expectations Figure 9 Pandora Radio Controls - Similar Artists & Feedback (Highlighted) Figure 10 Last.fm UI Home Figure 11 Last.fm - Listening report Figure 12 last.fm Song "Heart" Control (Highlighted) Figure 13 Allmusic UI - Home Figure 14 Allmusic UI - User profile Figure 15 Allmusic Add & Ratings Controls Figure 16 Cover Art Style Elements Figure 17 Global recorded music revenues - source: ifpi Global Music Report 2018[37]. 67 Figure 18 Use of streaming services - source: ifpi Global Music Report 2018[37] Figure 19 Discover Weekly feature and Spotify personal taste profile.[43] Figure 20 Genre view Figure 21 Search with related artists area Figure 22 Search with related artists area and info sign Figure 23 Agent options Figure 24 Scout Training View Figure 25 Scout Information View Figure 26 Scout Map View Figure 27 Genre view with marker List of Tables Table 1 Service Comparison V

8 AI-based Music Discovery Application Design 1 Introduction 1.1 Research Background The current climate of AI (artificial intelligence) development and research is favoring rapid advances. Emerging research in this field is fueled by technical progress and substantial investments in the design and development of algorithms and computing. As more and more resources are poured into this technical development, the capabilities of what is widely called Artificial Intelligence, used synonymously to Machine Learning, is increasing fast. On the other hand, Human Computer Interaction (HCI) with AI has not been explored thoroughly yet due to the short time designers and researchers could utilize the products of recent advances. But as we can observe more AI tools becoming available outside of technical personal, the necessity for focused research increases too. The areas of possible research range from concerns about ethics, responsibility and usefulness to emerging new services, user experience and interaction design. Currently artists and other creatives are on the forefront of experiments with interactive AI programs, often making them accessible to the public[23,24,67,68], but there is still a large gap between traditional HCI and emerging possibilities with AI applications. In this research, we will explore the options to utilize AI for the purpose of music discovery with a focus on personal AI. Personal AI in this regard refers to a user-centric and interactive AI that follows a humanin-the loop approach, creating a conceptual service that gives the power of AI in the hands of the user. Relevant research and developments connected to this study can be found in different fields. As the work will focus on user s music habits and 6

9 Chapter 1 Introduction how he can interact with AI including actions, reactions and benefits, a core area on the technological side is existing research in Machine Learning (ML) and Interactive Machine Learning (IML). When looking at music preferences, reflective practices and the user needs, there are various influences from psychological to professional development research. Especially in professional development we can find practices and processes that are meant for efficient application and might favor an adoption and symbiosis with AI technology. 1.2 Research Methodology This thesis explores the topic by first explaining why the field of music consumption and discovery is of interest in regard to AI applications. To understand the current state of music discovery, a survey of different existing digital music is performed. Following, an examination of existing literature surrounding the topic of music consumption and the underlying functions of music is conducted. After having established the music background of the research, we will discuss the state of AI, agentive technology and possibilities of using a human AI collaborative approach to the topic. The conclusion is done in form of a concept design for a collaborative human-ai music discovery service. The design process first uses in-depth interviews with various possible target users and the discussion of the results as a starting point, before positioning the service, detailing the customer journey and suggesting system architecture and interface design. 7

10 AI-based Music Discovery Application Design 1.3 Research Usefulness The aim of this study is to explore how users can benefit from AI technology that is reflective of an individual user and follows the users interactive input for music discovery. The research will try to find the implications that arise by making a single individual the focus of a specific interactive AI with the goal to replicate, represent and or reflect characteristics of this user while serving the user with the resulting behavior. One can say that any tool that a human creates is a reflection of human needs and with AI this goes much further. The reflection on human behavior is an inherent aspect of AI development. Reflecting on individual and collective human behavior is part of AI, as it is needed to build software which acts what humans describe as intelligent. Within the before mentioned goals, it will be necessary to determine what behavior, knowledge, believes or similar can be learned by AI and what are the necessities to do so, while on the other hand looking at possible applications and user benefits. Advancing knowledge of this field and understanding the implications this has for users and designers is important for improving the value AI can generate for users. As this involves working with AI to create a user centered application, some aspects of the project will help understand how designers could or should approach working with AI. Additionally, the work will be able to create awareness for this specific, humanistic aspect of AI in the design community. 1.4 Scope Music Discovery After the initial focus on possibilities of using AI for personal purposes, the scope of the study needed to be narrowed. Based on the 8

11 Chapter 1 Introduction findings regarding the need for data availability and human behavior/personality aspects that can be assessed, three possibilities were weighed. Mental health/ Self Awareness: A focus could be set on the interaction between the user and a medium using text or voice input. Gathered data could be confidently analyzed using different ML techniques since the formats are well established in the field of AI. In the end it was chosen not to continue with this initial idea. On one hand any practice that would dare to claim health benefits would need scientific backing that is far out of the scope of this study. At the same time, it was found that multiple parties, as the before mentioned replica.ai but also other companies have been making substantial progress in this direction already which would limit the possibility of adding new discoveries in the space. Gaming: As gaming has been a traditional field of application for AI, it is reasonable to assume that there are additional possibilities for further advances based on the favorable circumstances that can provide relatively simple user data in a simulated and therefore easier to handle environment. But the purpose of reflecting the user s behavior would be more reliant on the actual game than on the mechanics of the AI agent. It would be possible to use such as a premise for e-sports or game testing purposes but might prove narrow in possible solutions. Taste development: With the rise of the smartphone, the digital consumption of media has risen to new heights. Already digital technologies and the services provided on them, especially social media 9

12 AI-based Music Discovery Application Design services, have been able to capture a high amount of daily attention from people. Famously, all this creates data that companies, like those already mentioned as companies in the AI space, Google and Facebook among other. These are companies that act as intermediates between service/product selling companies and the platform users, serving them extremely targeted advertising to monetize their services which are otherwise free to the end user. This data ranges from the text input and browsing habits of people to video, sound/music and images uploaded and consumed by users to the way users represent themselves on the internet. While the end user enjoys a free service with some of his attention being directed to advertising, it goes to show how much value there is in user data. Especially on social networks users create actionable commercial data en masse. The users not only showcase their own tastes and personality through different means- directly and purposefully, but also add data and information by interacting with many other entities and services on the platform, from companies they like to artists or political parties and individuals they follow, shared comments and reviews. Hugo Liu demonstrated in his study of more than user profiles of the platform MySpace that this data can be seen as a taste performance of the user. Further can be inferred from his work that the interaction of the user with the platform can have transformational qualities since the public display helps in reinforcing a person s own notion of who you are and presents a space where experimentation with ones is represented identity is relatively easy and flexible. His findings also led to the decision to focus within this spectrum of taste performance on the aspect of music as he found that Music interests accounted for the greatest proportion of this vocabulary, with over 70,000 unique music tokens mentioned by at least two users and 15,000 unique tokens mentioned by at least 10 users. [25] 10

13 Chapter 1 Introduction Taken together, the high amount of data concerning music, and the general widespread involvement of people in this topic as past-time as well as the aspect of music regarding the personality identification of their personality makes it an highly interesting field for design research. As we discuss later on, while people in general nowadays create a high amount of data, music has a fairly open community with some services and many people sharing their information freely. 11

14 AI-based Music Discovery Application Design 2 Music Discovery Tools Survey With the high interest of humanity in the practice of creating and consuming music, the topic of music discovery has attracted much attention over the years. As music invites experimentation, and musicians often strive to differentiate themselves from those who have come before, new technology is often picked up by those musicians and experimented with. As with music generation, over the years also music discovery has gone through many stages. Only about a hundred years ago, recorded music was still in its early stages and most music was, or better had to be consumed live. At the same time, the consumer industry as we know it today was only starting to take hold, bringing such entertainment to the masses. The development of the radio and the gramophone was a revolution for music consumption. Suddenly people didn t need a live musician to listen to music, which made it much more accessible. Especially in the United States, where much of these developments took place, the industry changed fast. Now people in areas that never have seen a particular musician visiting, were still able to listen to them whenever they wanted. Radio especially developed as a way to discover new music and the radio hosts and DJ s became the facilitators of this. With their reach over the airwaves, they were able to influence tastes and the underlying music market. New music trends were influenced this way by local DJ s broadcasting what was happening to a larger audience. Subsequently to music gaining a wider foothold in the population and becoming an ever-increasing industry, new mediums emerged. Music magazines about specific subcultures and youth culture became a popular format and TV added visuals to music performance broadcasting culminating in the famous American MTV channel. Allmusic, a service we 12

15 Chapter 2 Music Discovery Tools Survey discuss in the next chapters, was one initiative that created a 1200 pages book, listing as many records as possible when it first came out.[58] Later on, with the rise of the internet, radio shifted to web radio and more and more informal news sources for music discovery fans and enthusiast around the world started sharing their tastes and finding online. Through the possibility to share music online in.mp3 format, often these websites and blog actually offered the music for download. Sometimes this was especially interesting since some music recordings were not available commercially, but widespread music piracy was also a result of the internet development.[58] While blogging and social media resemble formats that can be compared to music magazines and radio shows, we can also find other tools now that present new interactive ways to explore the breadth of music. and are three websites that feature a map like visualization of music. Everynoise in particular showcases a wide range of genres, their location on the map being representative of musical relations. The visitor is able to listen to one selected soundbite from each genre to get a better idea of what this style sounds like. Liveplasma and music-map on the other hand are asking the visitor for one artist to start from, and subsequently generate a map with the artist at the center and other related artists connecting to it. They both provide a very small subsample of the overall music landscape this way and users can explore this step by step by clicking on other artists, which regenerates the map with the new artist in the middle. While Liveplasma also provides sound samples, music-map does not. Other more playful ways to discover music on the web can be found to, as for example Spotimap, a visualization that uses the map of the earth as background and put songs on the map according to their mention of a geographic location.[59] These webtools 13

16 AI-based Music Discovery Application Design seem interesting and playful in general but seem to lack functionality to use them over a longer period of time for structured exploration. 2.1 Service Benchmarks As a means to better understand the status quo of music streaming and discovery, this study performs a benchmark analyses of four services Spotify, Pandora, Last.fm and Allmusic. Each of these services will be introduced and analyzed regarding their backstory, customer promise, business model, features, data collection methods and interface. The services were chosen based on a preliminary survey of available services in which we looked for benchmark examples that represent outstanding examples for this study. Important was that the service had aspects and functionalities useful for music discovery and work with data and digital user interaction, to inform as much as possible about the status quo in this field. The wider selection included services such as Youtube Music, Apple Music, Tidal, Deezer, Prime Music (Amazon). Spotify was chosen for their exemplary use of collaborative filtering, which resulted in features which have been highly praised by users and media, which is strongly in contrast to reviews from Apple Music which is supposed to be lacking in this kind of functionality. Pandora is an outstanding example for their Music Genome Project, the self-proclaimed most comprehensive analyses of music ever undertaken [51], as well as their overall performance as one of the most widely used web radios, at least in the past. In the wake of the success of streaming service, many other webbased services have struggled to maintain sufficient user numbers to have a valuable business last.fm is one service that has been struggling with this but is so far still the class-example for music tracking and has 14

17 Chapter 2 Music Discovery Tools Survey achieved good integration with other services to stay relevant. Compared to other services, last.fm actually shares user data, and statistics, not just for the user directly but also for research and third parties, which makes it another choice for the evaluation. Allmusic is another interesting case that actually has its roots in preinternet times, as we will explain. With its mix of online database and music-blogging/news with expert curation and comments, it presents another type of service that makes it standout from purely databasebased services Heuristic User Interface Evaluation For the evaluation of the music discovery functionality, we will follow the guidelines for heuristic evaluation as suggested by Nielsen[69] and more recently discussed by Wilson[70] as this method is suitable for the circumstances and provides a flexible framework to tailor the method towards the study s need of evaluating specific elements without access to actual user reviews that are comparable. We will further focus in general on a task-based approach, centered on the music discovery features of the services while paying additional attention to the interface elements used for user feedback towards music within the service. The evaluation will be task oriented and the scenario will, adhering to the research topic, revolve around music discovery. The reviewer will in this case log into the applications and try to search for a new artist or song similar to another artist which the user already knows. We will use the band animal collective as reference, a band that is fairly known but not completely mainstream. If the reviewer succeeds in this, a secondary task will be done which is to add this new artist to the user s personal collection, 15

18 AI-based Music Discovery Application Design or otherwise try to bookmark or highlight it for later recollection. The test will be conducted on the desktop versions of the applications for reasons of comparability. The actual heuristics of the evaluation have been distilled from the literature [69,70,71] in light of the focus of this study. As the exploration of the applications is focused on a specific feature only, we are not concerned with such aspects as robustness of the system, error management or compatibility issues, which are necessary for evaluations of the quality of a working system, but not decisive for the use of a single feature. Although, if a larger problem becomes apparent which noticeably influences the performance of the feature, it would have to be remarked on. The following heuristics were chosen to represent a baseline understanding of what the digital services provide and how they work. Since this study aims to compare different services and gain a general understanding of the workings of these apps, while a more in-depth discussion is held separately on each of them, we are settling on the following four heuristics for the task-based evaluation and comparison. - Ease of use: Clarity of interface and associations on the path to accomplish the task - how fast and instinctive can the user grasp what is happening and make further choices. Is there guidance needed and or provided in difficult situations. - Transparency: This heuristic aims to give a measure of how transparent and understandable the workings of the app are that lead to the presentation of certain content and other performances. 16

19 Chapter 2 Music Discovery Tools Survey - Adaptability: To what degree does the application/system adapt towards the user behavior and the user have possibility for personalization of the content or interface. - Effort: How much perceived effort is needed from the user to arrive at the set goal. In addition to the UI evaluation, based on the discovery task, the services will be evaluated in terms of performance regarding: availability of Discovery Streams - the different ways to discover music within the tasks ramifications, and the perceived quality of the recommendations received. The evaluation will be performed by the author after having familiarized himself with the applications. Therefore, this will not be a case where first impressions can be recorded. Following this, the evaluation will not be concerned with initial learnability factors but more interested in estimating how a regular user would perceive the mechanics of the service. To follow the evaluation choices, the study provides a walkthrough with the evaluation. In addition to the separate evaluations, a summary will be provided at the end of the chapter for easier comparability for the reader Spotify The streaming platform Spotify has been one of the few tech unicorns coming from Europe in the last years. This title is given to - mostly internet- technology based companies that have been able to grow at an astonishing rate and disrupt whole industries. Just recently, this company accomplished a sort of economic crowning it IPO ed 17

20 AI-based Music Discovery Application Design successfully at the New York Stock Exchange. Spotify currently has a reported revenue of more than 4 billion USD and a market cap of around 28 billion USD with about 170 million users of which about half of them are paying subscribers.[45] Since this development is only a couple of weeks old, it is interesting to observe what is happening at the moment. So can we see, that as Spotify, for the first time, had to publicly announce their economic development numbers and actually disappointed investors partially, the company only a few days later announced changes to their service. Namely to offer more functionality to their free to use accounts, lifting some of the restrictions that were reserved for paying members before.[46] According to the company, a more attractive and free entry level will help the company to turn more overall users into paying subscribers in the long run. At the same time, a higher overall number of subscribers makes the service more attractive for advertising and other services, which at the moment is only a very small revenue stream for Spotify, which has been criticized from investors. In general, it can be said that Spotify follows a Freemium business model. It offers a free entry level which is limited in functionality and supported financially by advertising, while the main profits are generated by a paid membership model. Once a user has been convinced that it is worth to pay about 10 euros per month for the service (approximately 75 rmb), they then unlock extra features for their account, transitioning relatively seamlessly from one to the other. Advertising breaks while listening vanishes and such features as downloading songs for offline listening within the Spotify app. In addition to the standard premium subscription there are a couple of possible discounts like for family accounts or students. Overall a premium user has access to the same 18

21 Chapter 2 Music Discovery Tools Survey catalogue of music. Although the user is never able to extract a song from the app completely. Which in the eyes of some critics is a concerning development of digital goods, since a company might vanish any time for numerous reasons and could leave users with nothing after years of paying for the access. The main attraction of Spotify has always been their wide selection of music. Spotify managed as one of the first companies to cut deals with the major music publishing houses to bring their music to a music streaming platform, who have been refusing to do so for years prior. Companies like Napster have famously attempted this before but failed. Still today, with many other services presently trying to corner the music streaming market, Spotify is said to have an advantage over others through a catalog featuring more indie and alternative bands- referring to a genre of music that is widely popular overall but includes many bands that themselves can have a very small following. Although with more than 35 million songs in all the major streaming services nowadays it is difficult to compare this quality without generalizing or selection of very specific genres and sub-genres.[47] Features As discussed, Spotify s main functionality is the ability to browse a library of more than 35 million songs and play the music on-demand, which is streamed from the server to your device in that moment. The user can utilize different devices like a pc/mac, smartphone or even other devices such as smart speakers or gaming consoles. Spotify can be accessed from multiple devices and features native as well as browser- 19

22 AI-based Music Discovery Application Design based access. Therefore, any user can log basically into any device and use the web browser to start streaming. Additionally, Spotify has partnerships with different companies for example it connects to Tinder and Facebook, enabling the social sharing of music preferences but the company also works with Uber, enabling to hear your own playlist on a taxi ride. Apart from the standard search and play functionality, Spotify lets you collect songs in playlists, which is the preferred way for many users. The service created a renaissance for the concept of playlists in this way. Users can share the playlists publicly if wanted and invite friends to edit playlists together. For the social aspect, the app lets you follow other Spotify users or artists. In the beginning of the service it facilitated this network functionality by having users sing up with their Facebook account and replicating their friend s connections within the Spotify network. A user can then check the public playlists and recently played artists of any other public user or friend. In terms of music discovery, Spotify facilitates multiple way to find new music. 1. Social. As already mentioned before, Spotify enables users to follow other people s playlists and even shows the songs friends are currently streaming prominently in the desktop interface. 2. Browse. In the navigation panel, a section called Browse can be found, this leads to an overview site which features a selection of Spotify curated and created playlists that user can listen to and follow Figure 1. These range from country specific charts to new releases to genre playlists to playlists around specific events happening in the world or season. Within the Browse section, next to the general playlists that are similar for the broader audience is the discover section. In this section Spotify presents 20

23 Chapter 2 Music Discovery Tools Survey the user a personalized selection of artists and songs which are recommended based on past listening behavior. Therefore, the recommended songs and artists come with a tag that says why Spotify recommends this to you, mostly by naming the artist that the music is related to. Besides this feature within Browse, there are categories for podcasts, and local concerts. 3. Discover Weekly. Also, in this section fall the two famous Spotify playlists Discover Weekly and Release Radar. The former has been especially lauded extensively for their well-received recommendations. Each week a new playlist is generated based on your preferred music. While Release Radar additionally only features recently released music. 4. Radio. While Spotify does not connect actual radio stations, it adapted the concept of radio in the form of on the fly generated music playlists. A user can choose a single artist, album or playlist, or alternatively a genre and hit play. Spotify will then play a constant stream of related artists. While the user can start a radio in this way, Spotify also automatically starts playing in this mode if a user is finished listening to any chosen playlist or artist, meaning for example the end of an album is reached the app then does not stop but simply starts to play the album radio. In addition to the features in the app, Spotify makes other features accessible in the form of websites in a more experimental fashion. For example, while the standard application features almost no statistics of a person s past use of the platform, Spotify offers the website which can show a highly curated summary of the user s profile, pointing out the top artist and top track and preferred listening times in a day plus the top genres a user listens to. While the functionality is meant for users to enjoy some insights, it is also quite obviously meant 21

24 AI-based Music Discovery Application Design to attract advertising companies. The website even directly shows Spotify for brands as the logo and finished ever page with a notice that Spotify can help with building targeted marketing campaigns. Other initiatives like this are for example yearly summaries (year in music) and Spotify SweetSpot, which was a marketing related feature for valentine s day and would create a custom playlist by connecting two distinct artists through their common related artist and tell you the degree of separation.[48] Interface For the analyses of the interface, we will focus on the desktop and mobile app. Desktop: Spotify uses a dark theme interface which looks a bit cluttered as can be seen in Figure 1 and Figure 2. It can be separated into 4 different sections. Going from left to right, the desktop app features a section for navigation. On top are the categories Browse and Radio that host the features we discussed before. Underneath is an overview of the Library categories and every playlist a user has to scroll through. While all the other elements in the interface or more or less static, the main middle area changes according to the selected feature/section/category. In figure 4, the Browse feature is selected, while in Figure 2 an artist is selected. After searching for an artist or through other means, songs/albums/playlists are visible in the middle section and can then be either added to the collection through clicking on the songs and selecting the function, or by drag and dropping the item into a playlist/library category on the left. Additionally, on top of the middle sections respectively left and right aligned are the search input and the access to the user account. 22

25 Chapter 2 Music Discovery Tools Survey On the right side of the window is the section for Friend activity which features a more or less live update of your friend s activity showing which songs the person listens to. Each user is able to set a profile picture which is visible in this friends feed. At the bottom of the window is the music player section, it presents the user with the current playing title and the necessary controls. Depending on the mode that the user is in playlist, radio, discover weekly, these controls slightly change to feature different functionalities. Namely for radio and discover weekly, it enables to user to give the aforementioned feedback towards the songs. Figure 1 Spotify UI Browse 23

26 AI-based Music Discovery Application Design Figure 2 Spotify UI Artist Mobile App: The Spotify mobile app follows the design of the desktop app- Figure 3. It opens to a screen that features the similar functionality as Browse on the desktop with carious curated playlists on the top part of the screen. The lower sections are dedicated to the music player controls and the navigation panel. Through the navigation panel the functions Browse holding the full functionality like the desktop section, Search, Radio and Library are accessible. Only when the currently playing song is chosen, does the navigation panel disappear. Then the user has to go back to get the controls back, as seen in Figure 3 on the right. In terms of functionality one main difference of the UI s is that the social section with the current listening of friends is omitted. The user can still access friend s profiles through the search function. 24

27 Chapter 2 Music Discovery Tools Survey Figure 3 Spotify Mobile App UI, Home screen (left) Artist screen (right) Heuristic Evaluation & Walkthrough Artist Discovery Once the user is logged in, he finds himself on the Browse section of the applications, which serves as a collection of music recommendations from different angles, as described earlier. In our scenario, the seemingly most direct way to arrive at a specific artist would be to use the search bar in the upper part of the applications. In this way it would be a single click plus the typing of the first letters. Halfway through typing the name, the actual band appears as suggestion- top result. As the band is not necessarily the most famous band with these letters, it seems that the application favors music which has been listened to before or is part of the user s playlists. This is later subjectively confirmed by running similar search queries with much less popular bands and having the top result 25

28 AI-based Music Discovery Application Design refer to bands within a user s collections as top result repeatedly. Another click brings us to the artist page of Animal Collective. If it wouldn t have been suggested immediately, the user would have only had to finish spelling the full name to see the band suggested as top result. This works very well with artist names as they are often unique enough and ranked higher than songs. At the artist profile, we actually immediately see the rubric of Related Artists, where we can explore artists that are apparently similar. This section is positioned prominently on the right side of the top songs of the artist. Unfortunately, in this view there is no other information about these artists and they cannot be directly listened to meaning that if we would like to listen to a song from this artist, we would have to click on the name and it would take us to that artists page. Though in this way it starts to generate a lot of clicks for the user if he has to go back and forth between the original profile and related artists. At this point there are four similar artists represented for Animal Collective. It seems this is the situation when an artist has a Merch section in their profile. Merch stands for merchandise and currently shows mostly different albums of a band in a physical vinyl format, ready for purchase. In Figure 2 for example, in the case of the artist Mr. Scruff, this Merch section is absent and instead shows a slightly different artist overview which includes seven similar artists. Alternatively, the user can choose to click on the tab related artists, also visible in figure 2. This brings the user to a wider range of suggested artists. It starts with the same 4 that were already visible in the overview but continues to list 20 altogether. In this tab, the user can actually listen to the artists directly, without leaving this view. The user is actually presented with three buttons, overlaid over each artist. Follow, Play and 26

29 Chapter 2 Music Discovery Tools Survey More. It is not clear what concrete effect it has to follow an artist, nothing immediately happens either apart from a change of the icon when this is clicked. Play presents a clear command which starts playing the Top song of the artist. It is actually also not clear why a song is ranked Top within an artists profile. In most cases it seems to coincide with the number of times the track has been played by all users collectively, but since there are some cases where this doesn t hold true, it is not completely understandable. One could guess that the recent number of plays is stronger weighted than past ones, favoring recent releases, which seems to fit observations, but is speculation. The third button More brings up a sub-menu with different options like starting an artist radio or sharing the information through different social channels. In our scenario, the goal of the task is to discover a new artist upon deciding that the user likes it and then add that artist to the user s collection. There are multiple options to accomplish this at this stage. By clicking on the former mentioned button Follow, the artist would be added to a user s library but actually the common way for users is to add specific songs of an artist to one of their playlists. To achieve this, the user would most likely choose songs from the artist after listening to them and add them manually. This would happen now as the user is exploring the similar artists one by one, a quiet time-consuming exercise with many manual decisions. Apart from this manual exploration, the user could also choose to start the bands radio. In this mode a temporary, custom playlist is created, as explained before, which consist of a mix of the original chosen artists songs with these similar/related musicians. The mix seems to be about 4:1 and the playlist would actually continue to play as long as the user desires as it is dynamically added on. Here again, the user would most 27

30 AI-based Music Discovery Application Design likely either directly find a song that he enjoys and add it to a playlist, or upon discovering an interesting artist, listen to that more specifically but ultimately performing the same action. In addition to the already mentioned possibilities to find related artists, also the about tab in the artist overview can lead to finding related artists. While the former mentioned related artists are merely put on the screen without any further explanation of how they are connected, especially since the word related can many more things than just a similar sounding artist, this tab shows two more categories with related artists within context. The first part is the written biography of the artist or band, wherein the user can often find the mentioning of other artists and how they have either influenced each other, collaborated or otherwise crossed paths. The second part is a list of playlists wherein the band can be found, and which mostly is mixed also with other artists, although it can happen that single playlists consist exclusively of the original band. Interestingly, Spotify seems to have chosen to favor their company curated playlists over user generated ones and for Animal collective this makes 3 out of 5 playlists. After double checking with a random selection of artists, this seems to be generally the case, with some bands having exclusively Spotify playlists in this section. Again, the user would have to click through the links to where they listen to a song and add either the artist, the album or the song to their library or playlists. Following a summary of the impressions regarding each heuristic: - Ease of use: The interface is easy to understand in general and invites certain actions like a quick click on the play button to play through the more popular songs. It becomes more cumbersome when a user is interested in more than a general presentation of related artists and the 28

31 Chapter 2 Music Discovery Tools Survey only way to explore is to actually listen through the artists work. Some of the functions like follow are prominently displayed are actually not clear in what consequences a follow action has over for example simply adding a song to the library. Also splitting functionality over several tabs and within similar information presentation having inconsistent interactions like the different possibilities of the features related artists versus the related artists tab makes, has a slightly inconvenient feel. - Transparency: Many aspects of the suggestions that are presented to the user are not clear. Spotify presents the user with final relationships but it s hard to tell if the artists share specific genre elements or other similarities that make them related. Overall they quite clearly are not random relations but can be generally seen as musically similar, but since this is not clear until either the user listens to them or reads their biography, this presents a small hurdle that puts the burden on the user. - Adaptability: The adaption of the search query results to the user s past behavior performs very well but results within artists are not. There is no indication if any of the shown artists could be more interesting to the user or other personalized adjustments. - Effort: There is a big disparity between the first moment a new artist is recommended, and the time needed to wade through different artists since no additional information is given that would help filter. We can see how many users would be happy to either follow the more curated ways to discover new artists through listening, or rather are open to spending the necessary time to be involved in the search, but the interface and functionality itself seems fitted to the former. 29

32 AI-based Music Discovery Application Design Feedback Collection As this thesis s focus is concerned with music consumption and discovery and aims to find a way of purposefully structuring the music discovery process through utilizing new technologies such as userreflective AI and AI agents, we want to point out the features through which Spotify, and lather the others, are collecting user information and preferences. Spotify collects all the data on what a person streams, but it only counts a song as listened to after more than 30 seconds have been played. If the user interrupts the song before, it is not counted.[49] As every play that is counted factors into a user s taste profile, exploring songs that a user doesn t end up liking is not clearly differentiated from songs a user likes as long as both are listened to more than 30 seconds which is rather short. The user doesn t have much agency over this in terms of controlled input. In this way Spotify actually forces the whole system to act on the user behavior rather than controlled input. Although one could argue that all this is controlled input anyway in a digital service. Especially moving songs to the library and into playlists is more reflective of what the user actually listens to than any inconsistent rating input. This is especially true for any normal plays, meaning every play outside of the discover weekly playlist and radio mode. For the normal plays, the system provides a button to quickly add the song to the library - Figure 4, while interestingly Spotify uses different feedback mechanism in each of the two other modes. In Discover weekly the user can heart or dislike a song- Figure 5. While the heart button gives positive feedback, the dislike button triggers a negative response and lets the user add if this applies towards the song or all of the artist s work. If negative feedback towards a song in 30

33 Chapter 2 Music Discovery Tools Survey this playlist is given, the app lets the user know that this song or artist will be avoided in the future for the playlist. Figure 4 Spotify General Music Controls (Highlighted) Figure 5 Spotify Discover Weekly Feedback Functionality (Highlighted) Figure 6 Spotify Radio Controls and Feedback (highlighted) In radio mode thumbs up or thumbs down are used for rating music played. All the liked songs are then collected in a radio favorites playlist for later reference- Figure 6. While of course Spotify collects every single played song in form of data, the lack of negative feedback for most of the listening on Spotify makes it hard to judge what Spotify thinks about your taste profile in detail. Even within the modes where it is possible to give feedback, it is hard to judge how this effects the overall process. Also, it is not really possible to tell Spotify that you like or dislike certain musical genres or elements. In general, the most important feedback for Spotify is the listing of songs in user created playlists. Spotify is not only able to know that the user has a heightened interest in a song because he put it in the playlist 31

34 AI-based Music Discovery Application Design but can now know which other songs this one is grouped with and make use of collaborative filtering. This approach is quite often used in platforms with enough users, for example Amazon recommends you possible other objects of interest based on how your purchase matches with other people that did the same. At the same time experts criticize that this leads to a circle of self-reinforcement where generally popular songs become more and more popular and recommended since every placement in a playlist is reinforcing the status in the recommendation algorithm, which can lead to a severe disadvantage for new releases and less known bands, favoring popularity over actual fit Pandora Pandora, or Pandora Internet Radio, is just that, an online service that acts as a web radio. Similar though to the Spotify radio feature, Pandora does not employ human radio hosts that guide you through their shows and playlists. Rather, software is in charge of choosing songs individually for each of their human users in real-time depending on their choices. To be able to deliver unparalleled performance of this recommendation system, the service has developed their own ambitious initiative to categorize music, called the the most comprehensive analysis of music ever undertaken. [51] by the company- the music genome project. Responsible for the start of Pandora is Tim Westergren, a musician who spend years honing his craft and working various music related jobs before presenting his idea to a friend who helped him with the business side of realizing it. Interesting for this thesis, the original circumstances of what made Westergren realize the potential of personalizing music was when he worked as a film music composer. Westergren needed to understand the film directors vision of what the movie and its music 32

35 Chapter 2 Music Discovery Tools Survey should portray. To do this, he started building musical Myers-Briggs profiles for the director, analyzing fitting musical pieces and understanding nuanced preferences and tastes. Through analyzing the music, he would be able to connect new music, new discoveries to pieces that have become before which then grew into the music genome project and Pandora.[54] This started already during the internet boom in 2000, at this moment called Savage Beast Technologies and listed at the New York Stock exchange about eleven years later, valued at 2.6 billion USD. At this moment Pandora already had almost a hundred million users.[52] Since 2013 though, the company has been struggling to maintain their early success and is now down to 81 million active users and a revenue of 1.38 billion USD.(data from 2016). Nonetheless, the company employs 2200 people in 26 locations.[50] At this moment in time Pandora is actually only available in the USA, having ceased operations in other countries like Australia and New Zealand. Compared to other Streaming services which boast more than 30 million songs, Pandora used to only have a catalog of around 2 million songs but expanded to a similar broad catalog when they launched Pandora Premium in Features Pandora had for most of their presence a very dedicated web player functionality only, as can be seen in figure 4- with the main feature of personalized radio stations starting from the users past listening behavior in general or specific choices of artists or albums. Pandoras claim of superiority over other, similar, services are based on a self-proclaimed superior recommendation algorithm. It currently gives you the options of playing a radio, exploring related artists and reading a small biography 33

36 AI-based Music Discovery Application Design about the artist. Pandora can be accessed from multiple devices, with the browser, desktop and smartphone being the main devices but also through products like Roku streaming devices. With the introduction of Pandora Premium the service moved more towards functionality similar to other on-demand services. In the wake of the premium service launch, the company made the following statement: Sequencing is such an important part. It s not just grabbing the right music to put into a playlist for you, it s also organizing it and sequencing it so it flows, Phillips says. Those are really important qualities when you want to have a listening experience that just works.[53] While most services calculate which songs you are likely to enjoy, it seems that Pandora expands on that by actually spending resources on getting the sequence right when songs are played. User Interface Compared to the Spotify interface, Pandora presents its users with a very reduced screen. Album artwork is prominently featured in the center of the screen and while the app uses a single strong color with slight white accents and gradation, the overall impression is a minimal style. The remaining screen space is taken by recommended artists on the very left part, a search bar in the center to and similar to Spotify and other music players with the music controls at the bottom of the screen. Next to the controls are also the feedback mechanisms thumb up and thumb down button. In the upper right corner, the user can visit his account. During listening, the user can choose to click on some of the possible information 34

37 Chapter 2 Music Discovery Tools Survey and explore other artists, whose information will then be displayed centrally instead of the album artwork. Figure 7 Pandora Web UI Heuristic Evaluation & Walkthrough Artist Discovery In Pandora, the user starts with typing the band, in our case Animal Collective into the search bar. He can then either start playing the artists station or go to the artists profile page. In the case of going to the artists profile, the user will firstly see the biography of the artist which actually often includes references to other bands which can be followed to discover new artists. Further, after scrolling down through other information about the artist, the user will find the similar artist category which seems to consistently feature six artists. 35

38 AI-based Music Discovery Application Design Similarly, if the user chooses to start a station with the original artist, he would see the information on Figure 1 and been able to scroll down from the station view during a song from animal collective and arrive at the same similar artist view. Although, it is important to note that actually the playing of the station in itself is to listen to and consequently discover music that follows the chosen artist in some way. Still, the rather short and lackluster presentation of similar artists casts an unfavorable light on the discovery functionality. In terms of the performance of the radio, Pandora actually provided a textual description of what the radio based on Animal collective will sound like and that the first song is an example of that, as shown in Figure 8. Figure 8 Pandora Music Style Expectations The actual songs being played on the radio seemed to be from a much broader choice of music, branching off to other genres, which in some cases was quite interesting, revealing some surprising choices, but in others seemed to go towards mainstream bands that were not reflective of the intended direction. 36

39 Chapter 2 Music Discovery Tools Survey The best way to accomplish the final part of the task, the digital notetaking of the new artist, is not very clear. It seems like the best way would be to start a new station based on the new artist, but at the same time the user is left wondering if this is the only way. This adds the artist in form of an artist station to a user s personal list of stations, but it feels a bit like a workaround for the original intention. Figure 9 Pandora Radio Controls - Similar Artists & Feedback (Highlighted) - Ease of use: The pandora application is heavily focused on the web radio functionality. Discovering an artist in another way is secondary to that. Therefore, it is not surprising to see in the execution of the task some inconveniences, as long as the user doesn t want to explore through directly listening to the radio. As the functionality is very focused the interface itself is easy to understand but some familiar aspects that are common in other music applications are missing that users have come to expect, like a way of collecting artists or songs for future reference. There were some moments when elements like a back button are missing and the user is required to click on pictures to accomplish that, which is not immediately clear when using it. - Transparency: The information mentioned in Figure 8 provides interesting insight in the Pandora music analyses but otherwise the user is again 37

40 AI-based Music Discovery Application Design mostly presented with particular results without much context explaining choices. The system collects likes/dislikes and displays them for the user when he visits a particular bands profile and is expected to act on those, but as with some bands radio suggestions, once the system is not performing well, it is not clear what causes the issue or what can be done to better performance. - Adaptability: The system is geared towards presenting the user with actual personalized content which pairs a starting point chosen by the user, like a particular artist, with the past behavior to refine songs suggestions. This in itself is a very adaptive system. On the interface we can find some adaptive content like next to the saved stations of the user were recommendations for other artists are made. Other than that, the interface is rather static as far as adaption to the user behavior is talked about. - Effort: As the exploration options of similar artists are limited by either the number of artists that are available, or the way of not being able to freely listen to songs, the tool is demanding more mental effort and motivation for a user to do it this way. Feedback Collection As Eric Bieschke, chief scientist at Pandora in 2013 already described, users will start with an artist, which is the main indicator, then, especially at the beginning of a person s use of Pandora, any interactions by the user will be matched to similar decisions from other users who have come before, trying to supplement data. Bieschke says that already three interactions can provide valuable insights this way. He further explains that besides choosing the artists, the thumbs up and -down (visible in Figure 9) and skipping a song- are the main feedback mechanisms. He 38

41 Chapter 2 Music Discovery Tools Survey adds that a thumbs down carries more weight than a skip, since a skip is ambiguous- a user might like a song but isn t in the mood or just heard it too recently. Pandora also acts device sensitive and adjusts choices based on the knowledge if a client uses a phone or is in his car.[55] In a more recent interview, Erik Schmidt, Senior Scientist at Pandora commented on the upcoming use of ML inside Pandora: One of the most fascinating areas of research at Pandora is Machine Listening. In the music domain, we seek to develop systems that are capable of automatically understanding the musicological content of an audio signal. These systems rely heavily on supervised machine learning, and Pandora s Music Genome Project provides the largest and most detailed corpus in the world for performing this work, spanning over 1.5 million analyzed tracks. As a result of this dataset, we have been able to develop incredibly rich and accurate machine listening representations. [56] Last.fm Last.fm is a service particularly geared to music discovery and tracking, different from the before mentioned services in that it doesn t actually host its own music streams. But users are able to connect their last.fm account to many of the other on-demand streaming services such as Google Music, Spotify, Tidal, Deezer, etc. Last.fm seems to try to aggregate the music people listen to from different sources this way and it is able to act as actual interface to the services. So can a user start a recommended song through the last.fm website in the browser and have Spotify play the song locally. Similar to Pandora, last.fm has a long history for a web company, with the start in 2002 in the UK. Although it currently doesn t host any songs, it 39

42 AI-based Music Discovery Application Design actually used to have a big catalog as well as a radio service. Changes like this have occurred over the years and are a testament to the difficulty in monetizing a music streaming service.[65][66] Features Figure 10 shows the desktop browser interface on a subscriber account. While most functionality is free, the paid subscription account provides an ad-free interface and extended statistic functionality. But in general, the experience is now vastly different between paid and free user. As visible in Figure 10, the main functionality and purpose of last.fm is music recommendation, as that is what the home page features prominently. Last.fm provides a mostly collaborative filteringrecommendation approach, which recommends you music based on other users listening behavior. It mostly recommends either an artist, an album or a track. Second is the tracking functionality, which can be seen in Figure 11. Last.fm provides a user with multiple statistics including: Nr. of songs per day/week/month, total listening time, count of unique artists/albums/tracks, most played genres, listening time of day, nr. of newly played artist/album/track, percentage of mainstream fit, as well as ranking in social leaderboards that track things like nr. of discovered artists and overall listens. Compared to Pandora and even Spotify, these social feature, like the leaderboards are more elaborate. Last.fm not only lets you connect with other users and follow them, but it also shows you other people, strangers, who fit your music, calling them simply Neighbors. They are shown with name and music you have in common to encourage you to check their profile to discover what else they listen to. Additionally, users can leave 40

43 Chapter 2 Music Discovery Tools Survey comments at the artist/album/track page. And can write messages to each other, check out concerts, charts and featured musicians. When a user searches for an artist/album/track and it is in the catalog, he will be able to see information and statistics about it, like how many listeners with how many plays the artist has on last.fm, what genres the artist belongs to, which are the most played tracks/albums of the artist, related artists and if available a short biography. The user is also free to add information. Additionally, the user can find photos of the artist, event information and links to YouTube videos. At the same time, all tracks on last.fm have a link to purchase the work mostly itunes, Amazon or ebay. In addition to the possible connection of services mentioned before, last.fm provides desktop apps and browser extensions for you to ensure that every song you want to track, can be added to your last.fm database. Interface The last.fm web interface follows typical web styles with the main navigation in the header, where also the music controls are located. The feel of navigating it is very familiar in this way but the overall side can be very crowded. Some pages are quite loaded with information, which makes it in turn hard to find at a glance what you want. 41

44 AI-based Music Discovery Application Design Figure 10 Last.fm UI Home Figure 11 Last.fm - Listening report 42

45 Chapter 2 Music Discovery Tools Survey Heuristic Evaluation & Walkthrough Artist Discovery Upon logging into the platform, the user finds himself at the home view, Figure 10, in this view there is a slight chance to actually encounter suggestions of the chosen artist, especially if the user has been listening to them recently and through last.fm or one of the connected services, for example Spotify. In the likely case this doesn t happen, as was the reality in this evaluation, the users most direct path to the artists profile would be to press the search icon in the navigation bar and enter the name of the band. After entering the name, the user is redirected to a new page showing the possible matches for the band name with artists ranking first, followed by albums and songs. While showing the artist as the first match, at this point the results for the artist name feature a lot of collaborations of the original artist with other bands, ranked by popularity. Also a possible way to explore related artists. Popularity on last.fm mostly refers to how many listeners a band has. In this instance, the collaboration is treated just as an artist, with a profile complete with written information, songs, other related artists and such. At the actual artist profile, the users are presented with a comparable large amount of information which can be used to facilitate exploration from here. The profile features associated genre tags, other users that listen to this band extensively, written information about the band with references to other musicians, as well as similar artists (4), all which lead to overviews or information of what can be considered related music. Though the four similar artists mentioned on the page are actually all projects directly from the band members either solo or as collaborations with others. The profile even features a button dedicated to play similar 43

46 AI-based Music Discovery Application Design artists a function in effect very similar to the Spotify artist radio or pandora artist station. In addition to these option on the main profile page, the user can choose to open the similar artist tab and finds there a page featuring 15 related artists, starting mostly with bands with direct involvement of the original band members but also others. It is worth noting that this is only the first page and the user can navigate through a total of 17 pages of related artists for animal collective. Once the user has chosen one of these options and listened to a new artist, last.fm automatically records the behavior and the user will find the band from now on in the history of their use-data. To emphasize a song that a user particularly enjoys, he has the option to heart a song as seen in Figure 12, which is similar to liking or upvoting actions in other software or digital services. In general though, a user is supposed to merely listen to songs to build up their library of listening history. This is emphasized in the way the service presents the users library, which is an actual tab in the user profile. In this library, the users listening history is visible and available to go through based on the amount the user listened to an artist/album/song, and in what time frame, but for example a simple search for a name within the library seems not possible, presenting a problem if a user only remembers partial information of an artist. Also, there is no grouping of artists in playlists or otherwise to create certain collections. 44

47 Chapter 2 Music Discovery Tools Survey Figure 12 last.fm Song "Heart" Control (Highlighted) Ease of use: With the many different categories and topics that can be explored, the interface and software system require learning effort from the user. It provides many different channels and sources of information that sometimes intrude on the actual task with some of the information seemingly being more for the sake of the information than for the users benefit. Transparency: Last.fm often notes on the interface that users are free to add information and make edits to the content. This can range from writing the description of an artist or album to adding tags defining the genre. While this favors adaptability of the system, relevant for the next point, this also shows the source of some of the data. At the same time, the user experience is often lead by choices on numerical data and statistics- like to show the popularity of artists songs in a timeframe chosen by the user and this information is usually provided, making it understandable how rankings and other metrics came to be. Adaptability: The aforementioned aspects of visible and dynamic numerical data and user input possibilities are part of what users can do to influence the system and shown information. The home page, Figure 10, for example is purely dynamic content that is generated based on the users listening behavior, similarly many other recommendation sections within the app use this. At the same time, last.fm has social features that 45

48 AI-based Music Discovery Application Design lets users comment on artists and albums, as well as editing information directly. This leads to a change in the pages character depending on what users do or write. Although thanks to the page layout, this is kept in check and information architecture-wise has a secondary position. Effort: While last.fm certainly presents opportunities to discover a new artist on almost every aspect of their application, the user is left with the feeling that quite some effort is needed to go through the different sources of the information, listen to different artists and make choices. There doesn t seem to be some funnel functionality that helps to weed through all the information presented, which can feel overwhelming. Feedback Collection Last.fm s most obvious mechanism is tracking the actual songs that user play. Every song is recorded as a scrobble in your profile. Since some users are very conscious about their statistics, there are actually third-party programs to help you add other music to your last.fm profile, because the homepage itself doesn t allow that. In addition to scrobbling a song, the only other positive feedback is to heart a song, as mentioned before and shown in Figure 12, the press of the button signifies that you like the song especially. Negative feedback is simply recorded by skipping a song and not possible otherwise Allmusic While services like Spotify, Pandora and also last.fm have a mass appeal and are used by people in many different countries, markets and groups, it needed an interviewee that is considerably more engaged with music discovery than the average listener to become aware of Allmusic even though the service is actually the oldest one of the four here 46

49 Chapter 2 Music Discovery Tools Survey discussed. It was founded in 1991 and originally relied on an actual book filled with music references and a CD with the equivalent digital information. Founded and run by extremely passionate people, Allmusic managed to create a database with over 1400 genres, aggregating music as well as the information about their circumstances and achieved critical claim to be a cornerstone of tracking the development of modern music with their breadth. Allmusic didn t just track other people s work but actively contributed in publishing biographies and information about created music. Features Allmusic combines properties of a music magazine with an expert catalog of music and related information. Users of the service can create a profile and start adding albums to their profile which then is used by the service to provide you with personalized recommendation. They can also search using all 1400 and more genre names, specify release dates, and filter by rating. The ratings are one major part that sets the before discussed services apart. Allmusic has adapted a classic 5-star rating method, where users can rate each album but not songs or artists individually. The general interface provides the user with the option to explore new releases, often accompanied by an actual review, a curated Discover section, articles, personalized recommendations and access to the user s profile as well as search function. In the artist overview, the user is presented with a variety of information about the artist or album ranging from an overview of the 47

50 AI-based Music Discovery Application Design discography to the biography to credits, awards and an extensive list of similar artists. Additionally, the user is able to see the ratings of the Allmusic experts as well as user ratings. And the website only provides the first seconds of a song with a link to buying or streaming options, but outside the original website. Interface Allmusic s interface, Figure 13 and Figure 14, is again very similar to the familiar style and layout of websites all over the internet. It uses an unobtrusive, but strongly contrasted black and grey color scheme with the navigation panel in the header. Profile, search and other site layout elements follows similar often used web patterns. One interesting UI choice is that the site presents a quick search bar as well as a link to advanced search in the main navigation reinforcing the notion that AllMusic s users are relying a lot on the database search. Figure 13 Allmusic UI - Home 48

51 Chapter 2 Music Discovery Tools Survey Figure 14 Allmusic UI - User profile Heuristic Evaluation & Walkthrough Artist Discovery After logging in, the user starts from the home interface. Within this interface, as can be seen in Figure 13, the search bar in the upper right can be accessed to search for the artist. A drop-down window suggests possible guesses during the typing. Half way through the word collective the system suggests that the user might be searching for Animal collective and a click on the suggested name takes the user directly to the artists profile. If the suggestion was not accepted, the user will, upon hitting enter, be directed to a page with search results from which to choose. On the artist profile, the second large category is the related artists category, similarly to what we have seen from other services but with the 49

52 AI-based Music Discovery Application Design difference that this time, the presented artists are tagged with either the label similar to or influenced by. This is the only case in the four surveyed services where context was added to the related artists presentation. Again, the user can also choose to click on the tab for related artists to be taken to a page dedicated for similar musicians and again Allmusic presents a unique split of related artists into four categories, adding followed by and associated with to the two already mentioned. The general presentation also follows a much more sober and factual approach in showing all the artists on one page in a simple two column list, while the other services have always put more space for album art or cover photos to add visual elements to the presentation. At the same time Allmusic also provides many contextual links like genre tags or references to other musicians in the biography which could also be explored to find other artists. Allmusic goes so far as to have searchable links through dates like a specific year or even locations like a city. While Allmusic shines with information and connections of the artists, it is unfortunately not ideal when it comes to actually listening to it. As mentioned before, the service doesn t offer real streaming service and can only provide soundbites from the songs. Once the user has decided on an artist, he has to arrive at either a specific song or album to be able to add it to either his general collection or a custom list. Both options are accessible through a button clearly placed next to the album name, as shown in Figure

53 Chapter 2 Music Discovery Tools Survey Figure 15 Allmusic Add & Ratings Controls Ease of use: Allmusic is not the most impressive visual platform but the general layout and text favored approach presents a less distracting interface that although filled with information still feels very manageable. As said before, the more list/text style of the interface provides the user with a view that requires much less clicking through pages while still having access to a large amount of information, when compared to the other apps. Transparency: Allmusic actually provides content talking about music which includes written pieces about albums or artists and an accompanying rating, which, with some effort, makes ratings very transparent. Similarly, as visible in Figure 15, also users can rate albums and have the option to leave written reviews. Adaptability: The service provides a tab with personalized recommendations but apart from this section, the general website is very static and doesn t react specifically to user behavior or input. Effort: As Allmusic is mainly a service for discovering music and musicians, the functionality to fulfill this task is very fitting and straight forward. Nonetheless, overall there is a hidden effort necessary on the side of the user since the actual listening has to happen on a separate device/service and 51

54 AI-based Music Discovery Application Design Feedback Collection As mentioned before, Allmusic has an emphasis on actual ratings, employing their own staff to provide opinions on as many releases as possible. The user therefore can give ratings to albums and while he cannot rate single songs, he can add them to favorite songs in addition to putting a whole album in either the users collection or a user created list of albums. 2.2 Summary The goal of this first analyses was to understand the status quo in the field of digital tools for music discovery. We employed multiple ways of gaining different perspectives on the issue and with the analyses of the four services, different patterns became visible. Table 1 presents a general overview of some of the key aspects of the portrayed services. This shows that although the services work differently in detail, there are some elements all or multiple of them have in common. All of these services are in some form platforms that attract millions of users, reminding us that one of the interesting things about this market is the enormous number of people that consume music. With user numbers in the millions, these services all follow a similar monetization model, namely to have a base service that is freely accessible but supported with advertising revenue and a paid premium subscription model which provides extended and ad free functionality. The advertising itself depends on the platform, Allmusic, last.fm and Pandora are selling space on their websites for advertisers while Spotify mainly has advertising breaks during listening. In addition to these revenue streams, 52

55 Chapter 2 Music Discovery Tools Survey the different services also are supplement this with selling either music or merchandising, mostly through affiliate programs apparently. An important aspect for this study is the knowledge about how the services are creating their recommendation systems and algorithms. All the services utilize their users in some way, either for collaborative filtering, one of the things Spotify is doing very well with their focus on playlists, or for rating and information input. This can be seen within last.fm and Allmusic especially, although also Spotify provides a way for additional custom input once users dig deeper through the application. On the other side, Allmusic and Pandora have extensive manpower employed in utilizing experts for music cataloguing, tracking and analyses. While this is impressive and has produced highly interesting data, both companies also pair their expert work with computing power to bring the results to users in a personalized way and to use the user data to refine results. We can see that a combination of human expertise and machine computing can produce great results, although the current survey cannot judge the superiority of any approach. 53

56 AI-based Music Discovery Application Design Service Spotify Pandora Last.fm Allmusic Offer Music Streaming Web radio Music recommendation & tracking Music recommendation & news Founded Users 170 mil. 81 mil. n.a. n.a. Data collection Main Recommendation technique Collection functionality Feedback Mechanism Main Business Model User created playlists User behavior Collaborative Filtering Playlists Library function Like/Dislike (only partial) Advertising supported Freemium with paid subscriptions User behavior Song analyses by experts Algorithm using the expert system (filtered by user behavior & collaborative filtering) User behavior Collaborative Filtering User collections User ratings Expert rating and cataloguing Expert System Connected Music through cataloguing List of User Stations Play History Song/album collection Custom lists Like/Dislike Heart option on Song/Album Ratings songs Advertising supported Freemium with paid subscriptions Advertising supported Freemium with paid subscriptions Music Streaming Yes Yes Through integrated 3 rd Main Discovery Streams Pers. Recommendations Playlists (Genre/topical) Radio Similar Artists Other user s (activity/lists) Table 1 Service Comparison Radio Similar Artists Mood/Genre party Other user s activity Pers. Recommendations Similar Artists Radio Advertising supported Freemium with paid subscriptions Through external 3 rd party forwarding Pers. Recommendations Articles Similar Artists Genre Exploration Similarly, important are the various Discovery Streams that can be found within these services. Discovery streams are various sources of information are inspiration to find new music. A classic, non-digital stream would be a trusted friend in the real world. Within these services, the trusted friend is replaced by other users or algorithms. One way or another, 54

57 Chapter 2 Music Discovery Tools Survey all services provide personalized recommendations. The common way seems to be either to straight forwardly present suggestions like based on your past history of listening to Animal Collective, we think you will also like XX, or to have a radio functionality where the algorithm chooses fitting artists after the user initiates it. Spotify and last.fm have both of these while Pandora predominantly employs the radio function and Allmusic is bound to direct recommendations. At the same time Spotify and Allmusic put significant effort into delivering also curated content in form of topical playlists and artist features. These three sections direct recommendations, personalized radio and curated lists and features are the main discovery streams. Although the different platforms are presenting their own formats and interaction patterns, the overall look and feel and behavior is comparable. One factor that contributes to this is a recurrent style element that all of these platforms have understandably adapted cover art and the nostalgic square format as was standard with CD s and vinyl s before. In Figure 16 we see snippets of all four platforms repeatedly featuring these square visual elements. But at the same time, the interaction design seems to generally be more reminiscent of familiar structures found for years in music players, file structures/lists and generally standard website layouts. 55

58 AI-based Music Discovery Application Design Figure 16 Cover Art Style Elements Overall this field seems to stay within well explored boundaries in terms of interactions but also seems to struggle with optimizing user interaction and the way to present all the available data while guiding users. Although last.fm and Spotify can be seen experimenting with more playful interactions in some aspects, the main parts of the applications stay true to established patterns. 56

59 Chapter 3 Functions of Music 3 Functions of Music When we think of music, the plain idea is that a person listening to it is doing it for the pleasure of listening. The actual use and function of music though is more nuanced. Merriam, an anthropologist discusses the various forms of music functions with describing ten different functions: The function of emotional expression; -aesthetic enjoyment; - entertainment; -communication; -symbolic representation; -physical response; -enforcing conformity to social norms; -social institutions and religious rituals; -continuity and stability of culture; -contribution to the integration of society.[35] These functions serve largely either personal or societal uses but show how wide the application of music is considered and how deeply engrained in various aspects of society and human life it is. Hargreaves discusses the function of music as well and, after surveying Merriam s definitions, groups and simplifies the definition to arrive at only three categories. He suggests that there are three overarching themes: the management of self-identity, interpersonal relationships, and mood.[32] In the light of these functions, this information helps to shape the reason why better music consumption would be beneficial to a person. It could be suggested that better tools to access this important function of music in our lives would enable a better performance of such and lead to better results in self-identity, interpersonal relationships and the mood of a person. 3.1 Music and Identity In the press release of his paper about insights into personalities through music preferences, Dr Jason Rentfrow, a psychologist from the University of Cambridge is quoted saying: music is a mirror of the self. 57

60 AI-based Music Discovery Application Design Music is an expression of who we are emotionally, socially, and cognitively. [26] The research itself goes on to show how certain personality attributes can be predicted by analyzing music taste. Their work examined influences on a person s leaning toward empathy and systemization while pointing out how other personality attributes have been already shown to correlate with musical taste in earlier studies.[27] One such venture was conducted by Rentfrow himself and Gosling, while at the University of Texas. The work is composed of multiple studies seeking to shed light on the supposedly neglected topic of music within social and personality psychology. The question they set out to answer was simply why do people listen to music? and within the research show that music plays a very important role in people s life and towards their personality too. They point out that Music is a ubiquitous social phenomenon. It is at the center of many social activities before stating Just as individuals shape their social and physical environments to reinforce their dispositions and self-views the music they select can serve a similar function. This goes in line with the before mentioned behavior of people selectively constructing their own persona in situations where they have a choice- like in a social network profile. Similarly, further on is added, that people use music as a badge to communicate their values, attitudes, and self-views and believe that the music they listen to says something about who they are.[28] So, in the act of choosing music itself, one might assume that people will consider how their choice might be seen by others. The dynamic of showing one s taste in music and being judged for it can be seen materializing, for example, in some features at Spotify. Spotify users 58

61 Chapter 3 Functions of Music would be by default publicly visible to other users, but since the service is aware of the stigma that might come with certain music, it offers the option to selectively hide a listening session from others. Important to note for this thesis is that Rentfrow and Gosling go on to say that Although music has enjoyed considerable attention in cognitive Psychology, very little is known about why people like the music they do. Following this up, they express that the research that exists mostly examines a very limited spectrum of music in correlation with a limited spectrum of personality traits, which taken together can seem to provide insights but isn t satisfying from a scientific standpoint. In the course of their study, they find similar results that link certain personality traits to a preference for certain music genres. For example, individuals that had personality tests showing as open to new experiences and with high verbal abilities were significantly correlating with preferences to Reflective and Complex music which harbor genres like classical-, blues-, and jazz music.[28] While the mentioned study was conducted in around 2002, Nave et al (including Rentfrow) recently performed a similar assessment using the latest digital means, namely Facebook, to reach a more general population. They largely confirmed the findings, lending more credibility to this line of research although they added that the circumstances for music preferences are very complex and could largely by driven by contextual and cultural influences which might dwarf the actual influence of personality traits on musical preferences.[29] Other studies come to similar conclusion throughout different age groups and social circles. Such as a study conducted with year-old students in the United Kingdom, which confirms the perceived importance of music also for adolescents and that they see music as a social badge.[30] 59

62 AI-based Music Discovery Application Design Reviewing these findings, we can understand that first of all music is important to people as a part of their life, but also serves as the mentioned badge as a social token which can be shared with others to communicate about one s personality and preferences, indicating a wider range of believes and personality traits than the literal music suggestion. Additionally, it shows that a link between personality and music preference exists when looked at societal levels. On the other side, the study did not provide insight into how music might shape personality. 3.2 Self-Reflection Dewey defined reflection in his book How we think in 1933 as active, persistent and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it and the further conclusion to which it tends. He continues to state that in reflection a person finds himself confronted with a given, present situation from which he has to arrive at, or conclude to, something that is not present. What is present carries or bears the mind over to the idea and ultimately the acceptance of something else. [7] Karen Mann et al summarize the state of professional reflection practices for application in healthcare, but applicable in general in professional practices as the following (edited): to learn effectively from one s experience is critical in developing and maintaining competence across a practice lifetime. Models of reflection include critical reflection on experience and practice that would enable identification of learning needs. As one s professional identity is developed, there are aspects of learning that require understanding of 60

63 Chapter 3 Functions of Music one s personal beliefs, attitudes and values, in the context of those of the professional culture; reflection offers an explicit approach to their integration. Building integrated knowledge bases requires an active approach to learning that leads to understanding and linking new to existing knowledge. Finally, taken together, these capabilities may underlie the development of a professional who is self-aware, and therefore able to engage in self-monitoring and self-regulation.[8] The interest in reflection for practitioners is not only a personal one but has been growing because of institutional demand for professionals to document their status and show that continuous efforts of education are made to ensure that their practice is up to current standards. Which is especially necessary for professions where ongoing progress changes processes for the practitioners. Hence why we find literature in fields like healthcare especially. This goes to show that when we talk about AI, machine learning and self-reflection have a lot in common. One needs the reflection of the other to find the necessary data and models to function, and both exist to produce something else through the data analyses, one way or another Music as Therapy Another use of music that is worth mentioning is the application of music in therapy music therapy. Since the discussed properties mentioned before can elicit strong emotional responses and be actively used as suggested to influence the mood of people, therapists have been able to successfully apply music with patients. In active music therapy, the patient creates music in various forms. Receptive music therapy can be used to set the mood in a session where the patient then is free to 61

64 AI-based Music Discovery Application Design associate feelings or memories with what he hears and talk about it, draw or meditate.[36] 3.3 Summary This chapter explored the function of music for the individual who consumes it and examined different aspects of how other practices from the psychological field might help to facilitate music consumption in a proactive way that not only helps with the consumption per se but in the process with the individual s self-development. As we can understand from the idea of self-identification, - reflection. and -development, we can aim at playing a positive role in the development of musical taste. Although it has to be noted that to say a service can help build a better taste is a strongly subjective statement, but we do want to express that the aim would be to build a better tool for music exploration that limits a person s development as little as possible and rather supports it as much as possible, as a tool is always a form of support as well as limitation. At the same time, reflecting on the psychological aspect of music consumption brings to light the big responsibility and agency that people are handing over to algorithms. As automated algorithms are deciding most of what a person listens to, there are many problems arising, such as the issue of reinforcing loops that keep a user within certain categories because of the effect of being send back and forth and tending to accumulate at single points rather than causing entropy. Therefore, a better tool does not only act as elevation from the status quo but also works actively against a negative development. 62

65 Chapter 4 The Influence of Technology on Music Consumption 4 The Influence of Technology on Music Consumption As technology enters our life, it often changes the way we humans behave. The democratization of technology that has been happening in the last century has allowed the average person not only to access more technological products but also other mediums and services that take advantage of that technology. For music, this meant that suddenly music was not bound anymore to the presence of a musician or expensive machinery, but television and radio paved the way to making the presence of music ubiquitous to almost wherever a human was present. Because of this, Sundin remarked already 1978 that this generation of children have heard more music in their young age than their entire lives.[33] Hargreaves et al (1999) go on to say that the influence of music on everyday lives is increasingly changing because of three developments in technology. 1. Access to music. Through the internet the user can suddenly choose whatever music he likes. It is not just the availability of some music, like on radio, but to whatever music the user wants to listen to. With this change, the user selects the moment and the music to a degree of personalization that has never before existed in human history. 2. Miniaturization. The access to music in general needs technical devices. While at the moment in time, when these three points were made, mobile music players already existed for a while in forms of Walkman and portable cd-players, the moment of the ipod and other devices that had vast storages or mobile internet access was still a few years away, the miniaturization of music playing devices meant for people that the musical experience has been "individualized": it has become a soundtrack to everyday life, and thus a central part of personal development and identity for many people. And 3. The digitalization of music production. 63

66 AI-based Music Discovery Application Design The development of MIDI (Musical Instrument Digital Interface) and other technologies made it possible for music production at home and a change in the landscape of musicians while making production also more accessible. No longer is it necessary to be actually able to play an instrument to produce a song with it. Further, Hargreaves says that these three developments respectively lead to two changes in the music industry. First, as remarked before, the consumption is now curated by the user rather than a broadcaster. And secondly, boundaries of musical styles are becoming increasingly fluid.[32] These kinds of developments have been observed as well in other areas, with many of them obvious changes. It is hard to deny the power of change many technologies bring. While AI is being heralded as a game changer for many industries, the impact on a personal level is not widespread yet. Siri and Alexa might be the first tools that actually change some aspects of how people act, while other applications are more happening on the backend of technology, where the new technologies are optimizing algorithms for performance but are hard for people to understand their influence. For example, Spotify uses different Algorithms, including ML algorithms to recommend songs to their users within the service. Something that has been done with different algorithms for years, although not often have the reviews been as rave as for Spotify s Discover Weekly recommended playlist feature.[34] Kuijer and Giaccardi link the same kind of change to AI. The designers suggest that smart technology, agentive technology and similar, will have an impact on user behavior different from other technology. Compared to other artifacts, AI technology and design will need to be seen as a performer, similar to humans in some regard. And the developing performance of the AI will need to be regarded in the process.[31] 64

67 Chapter 4 The Influence of Technology on Music Consumption 4.1 Describing Music As part of the overall purpose of this thesis is to explore the possibilities of using data dependent ML in relation to this subject, it is of utmost interest that Rentfrow and Gosling discuss how to assess music preferences of people. In the before mentioned study, Rentfrow and Gosling not only link music to personality but in their study try to understand how to assess music preferences. In their work, they mention that there are many levels of abstractions theoretically possible in discussing music, and that they are focusing on the most common level in general music discussion between laymen which is supposedly concerned with overall genre and subgenre comparisons. Further on, they use a system of describing music with attributes and conduct tests in which participants rate a multitude of attributes after listening to selected songs. This was done to understand if a common denominator for music can be found.[28] Unfortunately it seems that they didn t challenge or compare the study with other methods. The music genome project, created by music streaming and discovery service Pandora, was an initiative especially concerned with this aspect of describing and categorizing music. The project aimed to create an overview over all possible music and hired musical experts to listen and categorize vast amounts of music. They created a system that provides 450 different possible genes for each piece of music. Experts then listen to the music and decide which gene a song has and to what intensity. A gene is an element that is part of a song like gender of the singer, use of guitar or other instruments, type of background vocals, etc. Pandora then uses this to recommend music to listeners based on finding similar songs.[41] 65

68 AI-based Music Discovery Application Design 4.2 Music Information and Big Data Although there are still some people that use and buy Cd s and a group of enthusiasts that revived the market for vinyl records, the majority of people listen now to music through streaming services. As can be seen in Figure 17, in 2017 the revenue from streaming services surpassed that of physical sales. Furthermore, Figure 18 shows that on top of user paid streaming services like Spotify, platforms like YouTube account for most of the time users spend with on-demand streaming services.[37] While YouTube is in general a video host and streaming service, it is often used for music streaming, not just with music videos, but a multitude of YouTube channels feature playlists similar to music radio shows and can be listened to with a static image as video. Greenberg and Rentfrow wrote 2017 a paper they called Music and big data: a new frontier [38], in which they look at how Big Data presents new opportunities for music psychology research. They point to the issue of past studies, which have suffered in quality because of small sample sizes and how digital means are able to overcome this through simple and economic access to users over the internet. Initiatives like #Nowplaying accessible on present a trove of data which extent only has become possible through these new internet technologies and social media platforms. The just mentioned platform continuously tracks the hashtags #Nowplaying among others on the social blogging platform twitter. The dataset has amassed 74,128,974 listening events, 1,537,602 tracks and 2,381,400 users so far.[39] For research purposes, this dataset and others like it, present the opportunity to get closer to a realistic portrayal of listening habits. 66

69 Chapter 4 The Influence of Technology on Music Consumption Figure 17 Global recorded music revenues - source: ifpi Global Music Report 2018[37] Figure 18 Use of streaming services - source: ifpi Global Music Report 2018[37] Although everything that is seen online, especially on any social media platforms, as discussed in chapter 3.2 can be seen also as a performance by the user, the data might be considered as more authentic compared to tests given to a person where he has to momentarily decide what he likes and give an overview of his preferences. Additionally, over time different companies and individuals have started to create knowledge databases about music. Allmusic.com for example is an online accessible database that actually started as a 1200 pages reference book in 1991 before moving online later on. It now has social features where users can collect albums present in the database into their user profile while the service suggests fitting music based on their selection. The 67

70 AI-based Music Discovery Application Design service contains about 3 million albums with about 30 million.[40] Similar databases and services are last.fm, musicbrainz.org, Soundhound, music genome by Pandora, the.echonest.com. All these databases are providing information about musicians and their work. This includes a variety of meta data which can range from song information, song texts, artist biography to concerts they attended, producer, label etc. Overall it can be said that the information that can be found about music and musicians is quite rich. As one can imagine, the more famous the artist, the more information can be found. Additional to factual information, there is also a second layer of third party information. One provider of such information are experts/semiexperts. This can be music critics, journalists, producers, or DJs/radio hosts as well as Music Bloggers. As music is such an enormous industry and past time for many people, with many individuals being very passionate about it, there are many people in certain subcultures or genres that exhibit the insider knowledge to make them experts within a very small domain. While some years ago the exchange of music information happened either in those small subcultures directly through physical presence at concerts and such, and the more public and professional journalistic writings in music magazines, ones a genre had enough audience, there was a huge growth in written material with the advent of the internet. Suddenly fans around the world where able to share their thoughts around music as well as the actual music through simply uploading it, often celebrating the exchange of more obscure music. These bloggers often curated music playlists similar to radio hosts but didn t present them live. Instead interested people could download the list along with the songs in some cases. Overall, these experts present what they know about the music, some factual connections, sometimes trivia, 68

71 Chapter 4 The Influence of Technology on Music Consumption and add their personal opinion why the music they present deserves praise or not. In the way that they put the musician and their music in context with others, they often perform categorization for others to reference. As styles and genres are very complex, it is mostly human experts that put them in a certain field. The Pandora Music Genome Project mentioned before was one of those especially concerned with categorization and differentiation. A second source for information is the informal presentation of opinions by listeners or fans. The online streaming services mentioned before like YouTube but also social networks itself not only provide the ability to listen to music or like them, but to talk about it too. Reddit, a news aggregation and online forum website/social media company provides a central platform for a multitude of different communities. On the platform any individual can make an anonymous account and create a sub-forum for any topic. 234 Million unique users visit reddit monthly. The music sub-reddit alone has 17 million subscribers and many other such forums can be found, often under various names that indicate the insider status of people. The forum for indie music for example is called indieheads. In these forums people discuss new releases, concerts, gossip about the relevant musicians and present music that they found and want to share with likeminded people. Spotify has revived the concept of playlists. While in the age of the ipod many users suddenly found that it works very well to simply but all the music one owns on the device and hit shuffle, random play, Spotify made it almost necessary to reverse this development. As Spotify presented the user with access to endless music and almost everything most people wish for, users started again to put their preferred songs and 69

72 AI-based Music Discovery Application Design artists into playlists. This made it possible for Spotify to mine this metadata, namely what musicians users group together the collaborative filtering approach. This way Spotify was able to relatively simply recommend other users songs that were often group together but that one users haven t listened to yet. Since this and other data Spotify gathers is a business advantage for them, the access to this particular information is very difficult. Spotify as well as Pandora don t openly share information about their users. Last.fm is the notable example, where access to general use data is available. They, Spotify, are not completely keeping everything secret though. In a journalistic interview for the online news outlet Quartz, some insights into the process were shared. The interview happened in response to a new feature Spotify released Discover Weekly. The feature presents each Spotify user with a personalized playlist every Monday. In the article the journalist receives from Spotify more information about what the company does with his personal data. They show that they create taste profiles of the user. In Figure 19 on the right, you can see the taste profile of the journalist which is used to recommend songs every week as seen on the left illustration.[43] Additionally, Greenberg and Rentfrow suggest that more technology like wearables will be another step to enrich data. Already now fitness and health wearables measure the pulse of people constantly. If done right, researchers would be able to correlate this data to other aspects of listening habits 70

73 Chapter 4 The Influence of Technology on Music Consumption Figure 19 Discover Weekly feature and Spotify personal taste profile.[43]. 71

74 AI-based Music Discovery Application Design 5 Human Augmentation and AI in Music Discovery On the first page of cognitive science researcher Margaret Boden s book about AI, we find the description: Intelligence isn t a single dimension, but a richly structured space of diverse information processing capacities. Accordingly, AI uses many different techniques, addressing many different tasks. [1]. The term Artificial Intelligence similarly aggregates many techniques, technologies and uses under one umbrella term. Boden further explains how in the computer science field, the very idea of AI is the foundation of the field, as Turing wanted to create machines intelligent enough to solve problems, which later resulted in his work on the actual first computer and his popular Turing Test definition for testing intelligence in machines. This helps to understand that throughout time the definition of Artificial Intelligence has shifted, driven by the dynamic perception of what makes a machine intelligent. Each time the capability of a machine to solve a complex problem is remarkable to the eyes of the public, it is attributed with being intelligent. Further on, when the use of such machines becomes commonplace, the colloquial nomenclature used in relation to the same machine is changed with the intelligence dropped. Nonetheless, under the term of AI, we find the sub-group of Machine Learning techniques and Neural network. Although the term only recently became commonly known, the field started similarly in the early 1940 s with the work of neurologist and psychiatrist McCulloch and Walter Pitts, a mathematician, who proposed the possibility of binary computing involving methods that relate to how neural synapses work.[1] Samuel, the man who coined the term machine learning in 1959, stated in his work: The studies reported here have been concerned with 72

75 Chapter 5 Human Augmentation and AI in Music Discovery the programming of a digital computer to behave in a way which, if done by human beings or animals, would be described as involving the process of learning. And foreshadowing future goals for AI, he added: Programming computers to learn from experience should eventually eliminate the need for much of this detailed programming effort. This was written in a paper which taught a computer to play checkers, with AI and gaming, now more than 6 decades later again being one of the areas where much of the machine learning progress is being tested.[2] And although the time since then has been called an AI ice age, Yann LeCun, head of AI research at Facebook and one of the core people behind General Adversarial Networks, goes on to say that it only seems like AI and ML research and applications recently gained importance again. In truth, much related research continued under different names throughout the last decades.[3] Summarized, Machine Learning itself usually refers to the process of computing data to find patterns and correlations between input and output states. The magic is found in that the software itself searches for rules and models to connect input and output, therefore obliterating the need for the programmer to define a lot of variables. Most of the time this lack of initial definition is surpassed with the amount of data a system can learn from. In the scope of this thesis, the context for AI purely refers to this technical use of the term unless stated otherwise Interactive Machine learning Within the constrained field of AI research IML is only a fraction of the work as the overwhelming majority of AI research is dedicated to advancements on the algorithms itself around data analyses rather than elements of user interaction. Nevertheless, there are multiple aspects of 73

76 AI-based Music Discovery Application Design AI and user involvement that are being explored ranging from how AI is trained with the help of humans, to AI applications that use and convert human input, to human augmentation with the help of AI. John Langford from Microsoft research defines IML with: The fundamental requirement is (a) learning algorithms which interact with the world and (b) learn. One of the IML methods he talks about is active learning. A method that lets the algorithm determine which of the examples should be labeled (understood) to advance the overall learning progress considerably.[6] In that case an expert, probably human, would then label the example and continue this circle of learning. Understandably, this refers explicitly to the learning part of AI, when talking about AI agents, you would not only need an algorithm that learns, but a modular construct that can utilized the learned. A. Holzinger, working on AI for health, questions in his paper about IML the current focus on AML (automatic machine learning), which is the prevalent way of using ML and includes most of the other forms of ML. He suspects that one of the reasons for it, is the lack of knowledge how humans in the loop of a learning algorithm can consistently outperform AML, but continues to say that he believes that there must be a benefit in using the acquired expert knowledge together with ML. He points out that the modeling of artefacts, self-referencing errors that amplify over time, in AML is a risk that could be negated with a human in the loop. This is understandable in that any system with limited scope should not be left to make major decisions if it cannot be guaranteed that data sets are perfect, as we can find in healthcare. Further, interesting for this study because of the reference for limited data, he implies that humans can be beneficial because they are able to learn patterns from very limited data, while ML 74

77 Chapter 5 Human Augmentation and AI in Music Discovery algorithms in general follow the rule of needing a sizable amount of data to start.[4] Amershi et al list the differences and advantages in some cases as the following: model updates in interactive machine learning are more rapid (the model gets updated immediately in response to user input), focused (only a particular aspect of the model is updated), and incremental (the magnitude of the update is small; the model does not change drastically with a single update). This allows users to interactively examine the impact of their actions and adapt subsequent inputs to obtain desired behaviors. As a result of these rapid interaction cycles, even users with little or no machine learning expertise can steer machine learning behaviors via low-cost trial-and-error or focused experimentation with inputs and outputs. In their paper, adeptly titled Power to the people:, they explicitly infer how an interactive application like this not only helps the system learn, but enables the person operating it to see results, learn what action and reaction produces and has in that way a two-sided discourse.[5] It should come with no surprise that such IML studies have already produced self-reflective behavior on the human side, even when it was not an intended goal. Rebecca Fiebrink, the creator of the Wekinator, a tool that aims to make IML more accessible for creative creators like musicians and artists, revealed that in her study the participating professional cellists gained a new perspective on her own bowing technique, when occasionally she discovered through consistently poor model performance that her training data was not as clear as she thought it had been.[9] 75

78 AI-based Music Discovery Application Design Agentive Technology Similar to how the Roomba robotic vacuum cleaner has been the prime example for the advance of robotic technology in private homes, the Nest thermostat made headlines in popular news media around the world when it found commercial success for bringing new technology into the home of people.[21] The appeal of the nest thermostat is a smart device, an intelligent thermostat, that learns from the users behavior as well as its environment and controls the room temperature through other devices like the heating system to such an extent that ideally in the end no user interaction would be required. In this sense, the user acquires with the Nest thermostat a device, an artefact, that is given agency in the environment it is deployed in. Agency, agentive technology, or simply an agent, all refer to an entity, a product or even a service that exhibits to some degree autonomy with no human interaction. It is given a goal and will perform in various ways leading closer to the accomplishment of that goal. Noessel, talking about the idea of agentive technology in the realm of products further points out that an agent is a piece of narrow artificial intelligence that acts on behalf of its user. Narrow intelligence means its limited capabilities of reasoning and stands opposed the idea of Artificial General Intelligence (AGI). A narrow intelligence can accomplish noteworthy tasks and learn one way or another from its environment, but it will always be limited to a specific domain and specific ways of input and output. The concept of AGI on the other hand is reminiscent of AI that is capable to perform just as humans would. Just as it is often portrayed in science-fiction where advanced AI robotic or otherwise are capable at human level intelligence. He continues to explain that in case of the thermostat, an AGI device will be making decisions about how best to use its resources to manage its task and communicate with its users, that is, 76

79 Chapter 5 Human Augmentation and AI in Music Discovery to create its own interface and experience. Designers will not be specifying such systems as much as acting as consultants to the early AGIs [20] The importance of the notion of agents becomes more apparent when looking into AI literature. Norvig and Russel, the writers of an acclaimed AI textbook lead their book with an explanation of intelligent agents right after the foremost introduction and state: An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through effectors. A human agent has eyes, ears, and other organs for sensors, and hands, legs, mouth, and other body parts for effectors. A robotic agent substitutes cameras and infrared range finders for the sensors and various motors for the effectors. A software agent has encoded bit strings as its percepts and actions An agent is autonomous to the extent that its action choices depend on its own experience, rather than on knowledge of the environment that has been built-in by the designer.. The textbook further provides a classification of different forms of agents: Reflex agents respond immediately to percepts, goal-based agents act so that they will achieve their goal(s), and utility-based agents try to maximize their own happiness."[19] 5.2 Current State Much of the current progress in AI is not only found in academic environments but being put forward by companies and individuals outside of academia. While artists are experimentally advancing technologies by pairing them with a more humanistic or playful viewpoint and purpose, tech behemoths like Facebook, Baidu, Google, Microsoft, Intel and IBM are 77

80 AI-based Music Discovery Application Design funding AI research and creating products that are leading the field not only in commercialization but also on academic and research levels. Partly the reason why companies are in such a position is because they are able to convince researchers to work for them with higher salaries as well as a more favorable environment to advance their research. This is especially the case for tech companies like Google which are in possession of the necessary data to fuel much of the AI capabilities and would be otherwise hard to acquire.[10] Google also offers a wide array of AI services, but at the same time provides an artistic and experimental platform for AI that features such initiatives as Magenta, a research project focused on music and art using Machine learning, utilizing their service Tensorflow[22], as well as open source, cheap and accessible AI hardware to facilitate creators use of AI in hardware projects[23]. Many of these are collected and openly shared on the experiments with google website.[24] Non-surprising, also what is known as the startup scene is embracing AI, although sometimes seemingly more for marketing purposes than actual utility. Because of the wide definition of AI, it is simple for companies to use the term which comes with high expectations from the public. Replika.ai, currently in public beta, is one startup actually using machine learning for the idea of creating reflective AI agents. This application is a mix of an interactive diary with the promise of creating an AI agent in the form of a chatbot based on the user s writings. The promise is that the user based chatbot would then be a technological mirror for the person and therefore present an interactive discussion partner. Although the overall benefit of the app currently is closer to a relative simple chatbot that helps people to reflect on what is happening in their life by 78

81 Chapter 5 Human Augmentation and AI in Music Discovery reimagining the diary in the form of a friendly conversation and motivational content in-between. While the original idea seems to have run into problems of actually fulfilling the advanced promise of creating a mirror image based on your input, the company further on used the gathered data to create an open-source chatbot engine that can exhibit different emotions in conversations which can steered by the developer. Another area where actual AI agents can be observed are video games. Since the invention of digital gaming there is the ongoing necessity of having simulated behaviors of in-game characters, which has produced a wide array of projects utilizing AI and alongside various studies thereof. One of the researchers writing about AI agents in games is Joanna Bryson. In her work about Behavior Oriented Design for AI, she outlines the current state of AI agents in video games and aims to extend it with the goals of optimizing development and various applications of AI including in video games. When we look at these AI agents in video games, the definition of artificial intelligence is used differently. Rather than referring to computing and analyses in the form of machine learning (ML), these AI agents are called artificial intelligence because that s their appearance for the user. They appear as characters in a game, trying to approximate how a natural being would act in a similar situation. As she states in her review of the history of such agents, these are often far from intelligent although they have been improving over the years. Nonetheless, in the game environment such non-player characters (NPC) carry the name of AI or game AI, even though a lot of times the depth of such elements are very simple scripts that react to a specific input with simple behavior. Still, the research that has been conducted shows models for building AI agents 79

82 AI-based Music Discovery Application Design that allow for more capable underlying technology such as machine learning to be implemented. Already in Brysons papers from more than a decade ago, she refers to the opportunity of implementing learn and search elements within her framework.[12] As computing power and access to ML technology has increased, games such as ECHO by Ultra Ultra[13], are utilizing the power of combining the established concept of game AI with emerging AI capabilities fueled by ML. In ECHO, the game uses the idea of reflective AI agents literally in creating NPC s that are copying the players behavior and turn against the player. In this way the player has to watch his own behavior closely to not endanger himself. Using the workings of Artificial Intelligence as game mechanic has also be done before. In 1996, Steve Grand published his game Creatures [14], a simulation game that attempts to create an experience in which you raise and train a pet-like creature. This life forms are born, raised and die in your view and can be taught to respond to written commands and perform certain actions that are supposed to keep them alive and to help them prosper into a small society. In the creation of these game of life style games, AI plays a central role in imitating what real life behaves like and how it learns from interaction. In conclusion, the literature shows that the reflective power of ML tools and especially IML tools are existing. Human interaction with AI can happen on many levels, from requesting and being fed with information from the background, or directly addressed in training situations, to the use of AI agents in games to create interactive and more personal and unique feeling games. Existing research seems to brush on these topics in many regards, but it can be seen that there is a gap looking at how this 80

83 Chapter 5 Human Augmentation and AI in Music Discovery can optimally be utilized and designed to serve users and to explore applications that can profit from this. 5.3 Collaborative AI Already music discovery is mostly a collaborative task between human and machine. Software facilitates online networks where users exchange their knowledge and referrals and other software with their algorithms help in distilling this information down to bite sized servings of music recommendations and distribute them to millions of customers of streaming services like YouTube, Spotify or Apple music every day. While technology has made much of the music world accessible, through the right use of new tools, people could be helped to explore this world more successful and more efficient. At the moment, as we have seen in the benchmarks, the process of music recommendations is done with centrally optimized algorithms. Users are served the results and have to take them or leave them. Although the companies are open in saying that they are experimenting with different strategies and algorithms that can yield different results, the end user doesn t have a choice within that. They can only change the service completely to find something that better fits their needs. To counter this issue and touch upon points that have been discussed in the conclusions of chapter 3, in this service, we aim to provide a collaborative approach, where the human user and the service complement each other. As we see in the interviews, people are looking for music that they like, but that doesn t mean that it is limited to very similar artists as they already listen to. To a degree, music enthusiasts of either category are looking to be inspired, sometimes even looking for serendipity. As technology is good in finding similar things but lack 81

84 AI-based Music Discovery Application Design capability in making a jump, the idea is to involve the user more, though there are many ways to do that, we see potential in using two layers of human-tech collaboration. The first layer is an interactive knowledge graph style visualization. Software will be used to create a human navigable map with different levels of resolution and the user can customize what kind of connections he wants to focus on with the software adjusting the visualization along the way. The user will be able to see not just the general overview of music genres and artists but will see their own corresponding history. After the service has gathered enough data, it will be possible for the visualization to also take time in account, so that the user can see in what direction his music preferences are moving. In this map, the user can set way point marker in different forms. He can for example color in a specific are to indicate to the system which music genre or artists he would like to explore and also note which he doesn t. Particular this negative feedback is something that other services are often lacking and what can make it very frustrating to users when they know what they don t like but have no real way of telling the system. Once the user has chosen a point to explore, which can also simply be his current state of music preferences, he can engage the AI- agents. AI agents in this case are based on reinforcement machine learning algorithms which are well suited to allow for a human-in-the-loop interaction. There will be a range of standard agents which are continuously trained on the general information the system has about a user, but as human tastes are complex, each time a user starts to explore a new area, he can choose to create a new agent for this exploration or employ a multi-agent strategy. A new agent would be released, so to say, 82

85 Chapter 5 Human Augmentation and AI in Music Discovery at a point of the users choosing, or if the user decided for an area rather than a single artist or song, then a random song inside that area would be the starting point. Two processes then start one is that the software analyzes the song environment and starts categorizing similar sounding songs based on the audio footprint, the second is that the agents starts playing the first song for the user, who is in that moment asked to rate, or give feedback. Positive feedback reinforces the chance for a song like this to play again, while negative feedback would do the opposite. Each time the agent chooses a new song, it makes a choice between many possibilities that all have various connections. Sound similarity is one of them, artist closeness, year of production are a few more examples. Each time the user gives feedback, the agent learns if the choice made has yielded positive or negative results and calculates based on this the next step. While it s possible to use a single agent, most likely it takes some time to train an agent to produce results that are considerably better than random choices based on artist relations. To overcome this, there are also two possible strategies. One of them is to constantly compare with other agents that have explored a similar space and test if following an already explored trajectory is working well also for this user. The other is to use the general taste profile agent of the user as a support, although it has to be tested if this can work in spaces that are very different from the general taste profile. We expect that the training process of the agent gives the user not just a more personalized exploration tool and result, but that the playful interaction of training something that represents your taste will create an emotional incentive for the users to spend more time exploring a new area. Even if they are not immediately finding fitting songs, something that 83

86 AI-based Music Discovery Application Design would often turn users away quickly without having really seen what a genre has to offer, the user might feel that progress can be made by continued training. Similarly, the user might achieve a feeling of accomplishment by having explored an area, as now they have a visual representation of their knowledge, supporting their impression of themselves to be knowledgeable about music Knowledge Graphs for Exploration While the choice for a knowledge graph style interface was influenced by some of the examples we found in the desk research, a further exploration of the topic showed indeed that Mapping can be a cognitively adequate tool for handling information on musical structures. Like with other abstract matters, the visualization and the interaction help users to create and use internal cognitive representations. In music there is a particular need for this because it contains complex non-hierarchical structures. as summarized by Weyde and Wissmann.[60] In the design of a knowledge graph system, the exploration is where the user mostly interacts with the data. For such an exploration multiple processes for searching and finding can be used. Mottin and Müller present as overview three strategies: Exploratory Graph Analysis an artifact is presented, and the software tries to match it properties to similar artifacts inside the graph, a search by example technique. Refinement of Graph Query Results Gradually working through the data with narrowing of results through additional queries. Focused Graph Mining Through interaction with the user a specific area is determined, so that computing power can be adequately used, as the search of all instances of a large dataset wouldn t be feasible in most cases. 84

87 Chapter 5 Human Augmentation and AI in Music Discovery Interestingly, Mottin and Müller state further on that to optimize the process, they think it is a necessity to take the user in the loop and at the same time scale to real world graphs And also comment on the possibility of using Machine Learning to create more interactivity within graphs.[61] Machine Learning Algorithms for Recommendation In Oramas et al s work they state that A common way of computing content-based recommendation is learning a function that, for each item in the system, predicts the relevance of such item for the user. The application of machine-learning techniques is a typical way to accomplish such a task. In their paper about music recommendation with knowledge graphs they also show that databases like the accessible last.fm database can be successfully used to construct a knowledge graph and employ it for recommendation.[62] One such algorithm that seems a promising approach for this use case is VIME (Variational Information Maximizing Exploration). Houthooft et al present a curiosity-driven exploration strategy, an algorithm which utilizes an agent who alternates between exploitation and exploration choices, as typical for agent strategies, but in case of the exploration choices will search for the possibilities which cause the largest disruptions to the current model, if successful.[63] Another example shows work related to using sound features of a song and their connection through human association. Mostafa et al propose a deep neural net that, based on these hybrid features, rank artist similarity successful.[64] 85

88 AI-based Music Discovery Application Design As further in-depth expertise would be needed to make a definitive choice between methodologies and algorithms, in the scope of this thesis we can only take inspiration from these possible options but cannot judge their definitive advantages over each other. Further research with domain experts will be necessary to make final decisions, but as we understand general requirements, we will focus on the interaction side of the technology and assume that with the here shown example feasibility has been proven, although performance fidelity is unclear. 86

89 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service 6 Design of a Collaborative AI-Human Music Discovery Service With the information and insights gathered from the research, the next step is to gather input from users and understand their point of view. This feedback will be used to position the service and create the customer journey, before suggesting a system architecture and fitting interface design. 6.1 User Interviews After the initial research and literature reviews, user interviews were conducted to get a better understanding about what people think about this space, to understand their goals and ambitions and habits. A qualitative method was adopted as discussed by Creswell[72], which is suited to understand a person s attribution of meaning towards a subject through the authors analyses and interpretation of the gathered data. The qualitative approach also allows more easily to pair the results of the interviews with other observations and findings and include them in the discussion, which is helpful for such an exploratory study as this one with an open brief. The goal of the interviews was to gain insights into the habits of people that frequently listen to music, to understand their process and the tools they use, as well as more information about how they think about the idea of taste development personally. The hypothesis was a) that these people use digital means to consume music and stay informed about them b) they see music as a part of their personality and c) that they see their taste of music as something that develops and that they are interested in developing it further. 87

90 AI-based Music Discovery Application Design In the selection of the participants, the main goal was to gain a majority of participants that show an above average interest in music and music consumption and ideally represent different angles of involvement. The choice of this was made to assure that the participants actually actively engage in music discovery as a practice. While this should be useful in gathering actionable insights, it also limits this part of the study to be applicable to a niche, rather than the general population and as will be suggested later should be broadened in the future with further research. The initially approached people were mostly referred contacts who were known to have a connection to music in the vein of being either musicians, music enthusiasts or in a professional position that put them in touch with the music industry. Out of around 30 invites that were send to possible interviewees, 12 successfully responded. Within the 12 respondents, only one of them seemed to be without a strong interest in music. The interviewees were first asked if they were willing to participate in the study and informed about the use of the study as graduation thesis within the field of design and its relation to music. If they agreed to participate, they were supplied with the questions and asked to fill them out on their own time, without supervision and return them. In the initial approach all possible recruits were asked if they prefer to do face to face interview, which would have been actually conducted over Skype or similar services or to fill out the document directly. All of the positive replies chose to fill the document. No deadline was given, but they were encouraged to reply within the next days. In the end it took between 24 hours to a month to collect the filled documents from the participants. 12 participants were interviewed with a mix of open ended and factual-quantitative questions, with no given choices. As the interviewees 88

91 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service filled a questionnaire, it had to follow a prepared structure and also openended questions were in general kept relatively simple. While this style didn t allow for adaption of questions during the process or detailed follow-up question during filling, it helps with minimizing other biases that can come from face to face interviews. The interviewees ranged from 25 to 34 years old with an average of 28. The nationalities of the participants were as follows 2 Italian, 1 French, 3 Chinese, 4 Austrian, 2 German. Which makes the sample mixed European with a portion particularly Chinese. All of them had at least an undergraduate degree from a higher education institute and are employed at the time of the interview. While all the interviewees listen to music, two of the participants are performing DJ s. One is performing as semiprofessional musician and two are performing as hobby musicians. One of them is involved in the music industry through music video and music news video production. 6.2 Questions and Answers This section will discuss the answers of the interviews. 39 questions were asked. The participants were free to skip any question they wanted and spend as much time on the process as they wanted. Five of the 39 questions are related to demographics, which were partly mentioned before and are not discussed in detail in the following. To arrive at these questions, one participant was chosen as trial. This interviewee received a preliminary questionnaire with slightly different formulation and groupings. Then after having received the first answers, a second round of additional questions were given to this same participant, who agreed to be available for follow up questions. After having received the second 89

92 AI-based Music Discovery Application Design round of answers, the effectiveness of the questions was analyzed and improved if possible or necessary. At this point the resulting questions were sent to the remaining participants. How much do you listen to music every day? All the participants except the one who was known to not have a higher interest in music stated that they listen daily to music. After excluding the outlier from the numbers, the average of consumption is between around 3.2 to 5.7 hours a day, as most participants stated a range of hours rather than a single number, although with a wide variance. One participant stated that due to time constraints he only finds about 1 hour a day to listen to music, while most others gave a range of time from 2 to 12 hours a day. One of the participants, a designer and part-time DJ said that he listens to music between 2 and 5 hours a day but additionally has ambient music playing during his sleep. Two more people stated that they listen to up to 12 hours a day, without sleeping, one of them is a DJ, the other purely an enthusiast. Has this always been like that? If, when did that change? The analysis of the answers for this group of interviewees shows three different stage for people in their listening habits. Though the answers don t specifically call out their childhood, it can be implied that they did not include their early childhood time. Disregard this time, the first phase starts with the teenage years. And lasts to the end of High school. Since all of the questioned participants have a university degree, this can be seen as a second phase. And entering the workforce starts a third phase. Only one participant stated that his listening habits did not undergo any change. Interestingly, different phases affected people in opposite ways. While it seems that some interviewees found themselves in jobs where listening 90

93 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service to music is possible and was for them a welcome way of adding music to their lives for many hours a day quite regularly, others suddenly had no more time for music for the better part of their day. Most participants said that their habits changed at one of transition of these three phases. As all participants have a university degree, the results might be different for other groups, although it can be speculated that still a split between teenage years and professional career would be expectable. How much of that do you consciously listen to the music, or have it rather has a background playing? As it seems that many people have music playing for hours during their jobs or other tasks, this question was directed to see if they are aware of what is actually being played or if they see the music more as pleasant background noise. Five interviewees stated that most of their time spent listening is conscious listening. With three of them being involved professionally with music. Five more said that they hear music predominantly as a background while involved in other tasks. And two said that it s evenly split between background music and conscious listening. How is this consumption split between playlists, artists, radio? Playlists are mostly self-curated lists of songs. Sometimes with connecting themes like genre or mood. Artists means the user is listening to a specific artist only, like listening to a whole album, or in streaming services to all of their discography. Radio can constitute of actual FM radio or, rather common these days, the listening to web radios, which might or might not be the same radio show on both mediums. Three participants are fond of listening to specific artists predominantly. And only one actually listens preferably to radio, while for everybody else radio 91

94 AI-based Music Discovery Application Design ranks third. The remaining interviewees create their own playlists and listen to them as first choice. How much of that do you consciously choose the songs? As users nowadays can choose a medium, but not the actual artist or song, like radio or lists from other people, this question aims to determine the preference of the interviewees in that regard- Self-curated or externally curated. The majority of questioned users, 6 of them, are choosing the music themselves. With one adding that they choose songs mostly from recommended lists in music apps while another person said that they particularly discuss recommended or random suggestions from apps/services. At the same time three more people stated that they choose about 50% of the songs consciously themselves. How much do you listen to music alone, how much with other people? Nine out of the 12 listen almost exclusively alone to the music. The two DJ s consider their time spent with an audience not part of the normal listening experience. Two of the musicians share some time with other musicians but still have most of the time by themselves. One of the participants stated that she does spent some time with her husband listening to music. Overall, we can say that overwhelmingly people listen to music on their own. How do you organize your music? Five use mainly playlists. Four by artists, one by genre and doesn t use a particular system since she purely uses the music app. One difference is that while most of the interviewees have moved to almost purely using digital music, without any physical 92

95 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service medium attached, two still have physical CD libraries which are also organized by artist. How do you find new music/artists? Almost all answers were mentioning more than one way of finding new music. When asked how the participants find new music, the most mentioned way is through friend suggestions, seven times. Followed by researching an artist which they already like and finding similar ones, four times. Looking at online suggestions in databases as well as using Spotify recommendations and learning about new music at live events was mentioned three times each. Other possibilities that were mentioned are: from tv and movies, through YouTube, radio, music critics and music magazines. Would you say your taste in music is very focused on specific genres, or rather broad? Seven of the interviewees stated that they see their music taste as broadly applicable. The others varied in their focus. One DJ particularly said that he is focused on five different genres while others don t see their taste as broad on a general level but still spread out. One stated that he like high quality music of any kind but specially dislikes bland pop music. How would you describe your taste in music? What genres are you into? When asked about the genres, all participants listed a variety of them. As follows in no particular order: Rock, Classic, Ballads, Soulful house, dub techno, idm & glitch, desert rock, stoner metal, sludge metal, psychedelic, acid rock, blues, folk, funk, Hip Hop, alternative, Americana, jazz, electronic, house music, drone, experimental, psybient, psytrance, dark Psy/high tech, new age, latin, punk, Drum and base, trap, Chinese classics, swing, gipsy, 93

96 AI-based Music Discovery Application Design RnB, deep house, synth pop. In general, it can be said that people seemed to be along the lines of Rock and Electronic overall, and not exclusively of each other, but more following specific subgenres of those, while some, 3, additionally appreciate also classical music. On a scale from not interested in music to - hardcore fan/enthusiast, where would you rank yourself? Most interviewees naturally assumed to rank themselves on a scale from 1 to 10 with 10 being the highest number. As the people in this survey have mostly been chosen for their known interest in music, the expected and confirmed outcome was a high rating. Interestingly also the outlier participant rated themselves with an 8- quite high. The average of the twelve is at 8.5 with one of the DJ s additionally stating that he is the biggest music fan and enthusiast he has ever met. Interestingly also other commented that they see themselves as highly interested but don t want to be seen as part of a fan movement or such, which is often associated with groups of people that follow a particular band. What have been major influences on your taste in music? The answers to this question are showing the complexity of the topic and how intertwined music to general life and lifestyle is. Almost every answer mentioned at least two different influences. Parents are mentioned thrice particularly as early influences while friends in general are mentioned most often. Additionally, people that have learned an instrument or played in a band mention this as factor. Several participants mentioned that the places they used to go and interests in the lifestyle like drugs and alcohol influenced their musical taste development. One participant mentioned 94

97 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service that video games where a major influence, while two others state one particular band as deciding factor. How did your taste in music develop over the years? Was there anything specific that triggered a change in music interest? This question yielded some very personal development stories and infers that people attribute their interest in music to other events happening in live and the subcultures they encounter throughout. For example, one stated that playing violin was a first cornerstone of her music development, later on in university she started to listen to rock. Another connected his interests As younger one nirvana, rage against the machine, guns n roses etc. à general interest in music. Then more into hiphop & reggae à snowboarding, basketball etc. Afterwards rather electronic music (aphex twin etc.) à Partying, dancing etc. This shows how he sees the music style in accordance to activities he was interested in at the time. Others state that the continued engagement with music, the experience they gather is helping them refine their tastes. One in particular uses the words I learned to enjoy some noise artists as well which refers to a music style where no particular melody or classic sounds are present but mostly electronic noise, a style that most people would find obnoxious. But he sees as an accomplishment, partly intellectual in nature, to be able to enjoy this variance of musical style. Another one, a physics PhD student, explains that his need to code for hours made him listen to electronic music as a means to help him concentrate. The girl mentioned before that listens to music with her husband, says that this relationship has been a major change in music for her, as through the relationship she entered in a different friend circle and started to listen to their music which 95

98 AI-based Music Discovery Application Design subsequently she adopted as her own. Only one of the participants explicitly stated that the taste in music didn t change. How do you think does your taste in music reflects on yourself? This question proved difficult for people to answer. With four participants not understanding the question to a degree to give an answer or not wanting to. Four more interviewees stated that they think the music they listen to only reflects little or not at all on who they are as a person. The before mentioned girl that had a strong connection to one particular subculture and their music says that in the early years she wanted to show that she is part of this group and fulfill the clichés of that group. Which meant wearing black clothes and other heavy metal related identifiers. Is there an influence of the music you listen to, on who you are as a person? While in the last question the answers where not as clear as hoped for, this question, asking for the influence of music on the interviewee as a person was more insightful. 6 of the interviewees stated that they think music influences them in some way. It was mentioned that music serves as inspiration in their life or that the artist and their lifestyle is inspiring. Another said that it helped them connect to different people, in that regard serving not only as the connector but consequently helping to make them a more tolerant person. While only two stated that they don t think it influences them. Do you want to actively continue to develop your taste in music? This question received an overwhelmingly clear response. 10 out of the 12 participants stated that they want to continue to develop their taste, additionally phrasing it in a way that emphasizes their commitment like: 96

99 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service that s a damn sure thing and 100% sure. Only the one person less interested in music stated that she is not sure refinement is necessary. Do you reflect on the music you are listening to? Also, this question had a rather clear majority with 8 interviewees stating that they do reflect on the music and only two saying that they don t. While the other two abstained from answering. Do you use any tools (digital platforms) to do that? Anything to track your music? Four of the participants say that they do not use anything to track their music while six of the others are using digital platforms YouTube, Xiami, Spotify etc. to archive their interests. Could you imagine that reflecting better on the music you are listening too would be beneficial? While this is a challenging question and certainly borders on being too suggestive, we can see that six people answered that they could see benefits in reflecting better on their music. Two stated that they don t know. While three said that they don t need something like this. It is worth mentioning that one Chinese participant had the following to say (highlights have been added): It was just curiosity for me at the beginning when I started to reflect on music, say, I came across a song that s very interesting and avant-garde, I was dying to understand its origin, I have to read the bio of the bands and the lyrics meaning, and know what instruments the musician have used these were what music nerds do. There are of course a lot of information and you learn much more beyond the music itself. And then later, I build up my own music database, 97

100 AI-based Music Discovery Application Design and I have my own music map that contributes to the growth in personality. It s a process of learning about myself, definitely beneficial. We can interpret from this and other answers that her interest in music is not just for entertainment but a vehicle to maneuver herself just as music can be a vehicle to communicate and express. Although her use of the words music map has been figuratively, it is interesting to see the concept of maps coming up again, similar to how Spotify uses a mapping method for taste profiles. What are you looking for when you try new music? Most of the answers in this section can be categorized as looking for pleasure and inspiration. With three pointing out that they are looking for an emotional connection to the sound. Do you have any troubles finding new music? Would you say it s easy or difficult to find new music? The consensus for searching for new music is that it s not too hard to find new music but it is laborious and time consuming to find good new music, especially when looking for lesser known bands. How much do you talk with friends/colleagues about other people s music? This aspect proved to be very polarizing between people who are involved in musical activities other than listening and the rest. While the DJ s, musicians and the video producer stated that they basically talk as much as possible about music with others, the rest of the interviewees, with the exception for the girl which bigger part of her friend circle is 98

101 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service centered around a common music genre, stated that they rarely if ever talk about music with others. In that conversation, what are the things you discuss about musicians/music? Over the range of the participants, one repeating topic is the quality of the music, another is new discoveries. The performing musician stated that she discussed technical aspects since that is relevant with her band while the video producer said that the discussions involve how to make better music and money with it. Only one stated that he discusses emotional responses with others. Which musician are you listening to the most at the moment? To understand how people actually discovered some artists, this question was asked as an opener, with the next questions probing for more details in the process. As the question asked the most listened to at the moment, unfortunately two answered that they cannot point at a single artist, but all the other 10 interviewees listed one or more artists. How long have you known/listened to this musician? The time frame of the answers ranged from too long to remember to half an hour. Three of the participants stated that they are still listening to music which they have known for years. Four more specified a time frame of less than a year, down to a couple days and the mentioned half an hour. How did you discover this musician? Two stated they discovered it from YouTube suggestions, three through friend s suggestions, one from a tv-show, one on a radio and one on a best of list on his preferred online service. 99

102 AI-based Music Discovery Application Design If you are using digital platforms, are you making use of the recommended songs features or similar? Curated playlists and such? Which features are you using? Eight of the participants stated that they use digital platforms and their song recommendation features. This shows that first of all people are using digital platforms for music discovery and secondly that it seems to be working quite well. It seems like as people are already using the platform, they sometimes want to listen to new music and are open to just letting the software create a radio, in which the software choosing songs based on the user s profile and keeps playing them until the user interrupts it. How do you like them? Do you have some feedback for these features? Anything that you would like to change or suggestions for new features to add? Two participants stated that they think that the recommendation algorithms are repetitive and show too often famous musicians. One stated that YouTube playlists are pretty okay sometimes itunes really sucks here. Which can be seen as an indicator that users are, or at least could be aware of performance differences in algorithms. He also suggests using sliders which could influence mood and complexity of the music you are presented while another participant says that suggestions are dangerous for the overall music taste. Is there anything else that you could tell me that might help me understand the way you listen and choose music? The last question in the interview was an open-ended question asking for free association to the topic and suggestions which could provide additional insights. One interviewee stated that once he finds a new song he likes, he will search 100

103 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service for every version of the song live, remastered etc. Another points out that she prefers radio because of the truly organic/coincidence quality it possesses and enjoys the ambience, opinions and humor in the process. One interviewee also said that he would be interested in a functionality that recognizes how the brain responds to music and lets choices be made based on that. After the initial round of questions, four participants that were available, where asked an additional question to understand the moment they find a new song. This question was centered around the actual actions that were taken once a new song is discovered that fit their taste. One said that he would put the new song on a playlist and move on, while another stated that she will, after finding an interesting song, start listening to other albums of the same artist to see if she finds also other songs of the musician enjoyable. Another one stated that he not only listens to the songs digitally but would immediately buy the actual CD s and such to add to his collection and listen to basically the complete discography to make sure he was fully explored the artist s work. The fourth person asked said that upon finding a song he likes, he would listen to every available version of that song, which can come not only from the original artist but also cover version or adaptations from other artists. Which in turn can lead to the discovery of new artists through that connection. 6.3 User Interview Summary Considering the original hypotheses, a) that people use digital means to consume music and stay informed, b) people see music as a part of their personality, and c) people see their taste of music as something that 101

104 AI-based Music Discovery Application Design develops and that they are interested in developing, we see that a) and c) have been confirmed. On the other hand, with b), the interviewees were suggesting that music can have subtle influences on their life, but they don t see their taste in music reflecting who they are. In terms of user s music listening habits, it can be understood that within this group of music enthusiasts, there is a split of groups depending on their involvement. People that are practicing music or are professionally involved, have a different relationship and behavior than those that only consume music. Although their overall time spent listening to music might not differ much, the mindset and environment are different. One major difference is that the former group has related social moments, while the others see it as an overwhelming personal activity. Overall though, behaviors can be quite complex with different variables influencing personal processes. In terms of interest, it is clear that the interviewees are very receptive to the idea of developing their music taste, to refine it, but not as much when the discussion was about reflecting on their taste. This implies that the people feel like they are well self-aware of what music they listen to but also think that taste development is something positive, leading them to positive and pleasurable experiences. One interesting behavior that could very well be replicated with an intelligent agent or incorporated in dynamic processes, is the exploring behavior once a song has been liked. In that moment, closely related music is often thoroughly examined by the users. 102

105 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service All of these questions have been helpful in getting an understanding of different personalities and how they consume, not just technically but with what mindset. The overall sample size has still been quite small, and it would be advisable to perform more research in the with a broader audience as well. 6.4 Service Positioning Through the interviews and the exploration of the benchmarks, it became increasingly clear that a new service in this space would have to differentiate itself from existing products. Not only is the market extremely crowded already for on-demand music streaming for the general user, but the existing companies are investing strongly in tuning their recommendation algorithms and general offers and the users on a grand scale are happy with what they are receiving. At the same time, the interviews showed that more enthusiastic users are increasingly sensible to the music they are being served like being recommended only popular music defeats the users purpose of discovery. These people invest a lot of effort on doing just that discovery but seem to still rely mostly on manual work to go through lists of songs and search in a way that is not ideal in these on-demand services. The services don t seem to actually facilitate the process that users are going through and rather promote their own simple and streamlined recommendations instead of giving more agency to the user. Which works incredibly well for the general population that uses these services such as most of Spotify s millions of users but is creating friction with prosumers.[57] Nonetheless, we understand that the unrestricted access to a music catalogue is still important and that it would be hard to compete with a service that might 103

106 AI-based Music Discovery Application Design be fun but doesn t have the actual access to a catalog like Spotify, Pandora or Apple. To tackle this problem, the here presented service will aim to be a tool that users can use in addition to other on-demand services. As Spotify and Last.fm for example provide API s, another company or service, such as ours, can take advantage of this and utilize their catalog. Similarly, one advantage of the music space is that many databases and services are available, if not always free. The core of the service will provide the user with a knowledge graph style visualization which the person can explore through manual traversing of the graph and add information like way markers while being aware of what are the areas and musicians that the user has listened to in the past. Therefore, the user is able to visually understand which musical genres are related and learn more about the musical scene overall. As is often the case, users are able to appreciate new works of art more if they are presented with more information, putting the work in context. As such, the service will aim to provide additional information through the factual knowledge that can be found also in other services, but complement this with secondary connections from databases, that show as links in the graph visualization. This can be shared concerts or social connections between bands, but also articles, among others. After considering the processes and circumstances that people create to discover music this service will consider an approach that lets it be a part of their memory. As so much of the media we consume flows through our devices, but comes from different sources, it seems it would be a great advantage to create an addition to the service that helps you to keep track of what you are hearing and then to facilitate the discovery 104

107 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service process through data visualization and the help of intelligent, machine learning fueled agents. The user would enable a custom app on their desktop and smartphone, which would keep track of what music a person hears. This doesn t have to be restricted to listening to musical works directly through other streaming services, but as the interviewees mentioned, can be part of a tv series or other video material consumed on the internet. The tool itself should also be a way to record more than just if you like or dislike a song or an artist. The knowledge graph allows to add information for future reference. There can be at least 4 categories of how a user feels about a song the standard Positive, Neutral, Negative, and in addition the status of Interested, which marks a song that possesses interesting attributes but doesn t fall in a category that would favor repeated listening from the user. 6.5 User Definition The target audience for our service is comprised of music enthusiasts that are either very passionate about music as part of their life but consume it mostly for pleasure, and on the other hand people that are actually involved in the creation of music either directly through the performance as musicians are in the creation of music experiences which can be as DJ, film composer or other music related professions. These don t have to be necessary people that earn their living through music but are at least seriously involved in the topic as a strong hobby and might or might not make some money from it. For the first category, one comment from the interviews can indicate more about the mindset of these users: Music taste is part of personality, 105

108 AI-based Music Discovery Application Design and the development of music taste reflects one s personal growth. I used to think I have an extreme personality, but when my music taste shifted in college, I started to complete myself by looking into different music genres and add them to my characters. This user group doesn t engage in creating music, but they follow the music industry closely and are genuinely interested in music as an artform and form of expression. They are not necessarily, and don t see themselves as fans of one particularly band but enjoy a variety of musicians and genres, although those might be grouped relatively closely. 6.6 System Architecture The overall structure of this service consists of multiple levels outside data(api), internal computing, user input through web interface/app, sound input from app (desktop, mobile). The central point for the user is either the web app on desktop or a dedicated mobile app, which will essentially mirror functionality and only vary in necessary adjustments for screen and interaction. Meanwhile the bulk computing will have to happen on company run or rented servers. Ideally a desktop- web app would offload some of the computing needs to the user s CPU, as computing ML is energy and processor intensive. This might even have to be accounted for on a mobile app, meaning that mobile functions might need to be restricted to limit computing intensive tasks. This scenario actually opens up an interesting alternative similar to how crypto-currency mining currently happens through browser on certain website, often more as an unwelcome abuse of an unknowing user, a desktop user could enable mining on his pc and earn a digital token in response to that. These computing credits could then be used by the same 106

109 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service person later on during his use of the mobile app. At the same time, a model like this opens naturally to monetizing options. Users that use more mobile computing than what they produce with their desktop could buy computing credits from other users or the company, while users that feed more in the system than they need are able to use the service for free otherwise. The outside data is received through API s (application programming interface), which are digital ports to other applications that are prepared for data exchange. These API s are not always freely accessible but for music there are various possibilities between free and paid. Possible API s for this service would be: - Shazam API for music recognition. Once a sound is recorded through the device app, it is sent to Shazam or similar which is a service that can reliably identify a sound and return the artist and song. - Spotify API for music streaming. Spotify provides an open to developers building app, this can be utilized to integrate a vast playable song catalog as well as data about similar artists and other relations. - API s for music information/metadata: Last.fm API/dataset for music data, relations, user ranking, etc.; Musicbrainz, Wikipedia, Discogs. When receiving the data, it has to be fed into the company s server and routed to the user where the data is received and presented to the user, waiting for the next interaction. These interactions work on two levels mainly - the knowledge graph used for visualization and direct exploration by the user and the agent interaction with the human in the loop. The following two chapters explain those mechanics. 107

110 AI-based Music Discovery Application Design 6.7 Customer Journey The customer can use the music discovery service in different ways as to his momentarily preference. Generalized, the main activities of the user are organization and passive or active exploration. In passive exploration, the user relies on the software s recommendation and only provides feedback if he likes it or not. In active exploration, the customers make use of the map tools and works in collaboration with the software, seeing how the recommendation through the general system as well as the agent is providing suggestions, as well as navigating the map himself to add points of interests or directly check out artists. In the following we discuss a possible exemplary customer journey covering main features. In this scenario the customer is already a user and familiar with the working of the service. 1. The customer uses his desktop pc to stream a tv series online. As the user has installed the browser extension or desktop app before, the software is able to analyze the sounds coming from the video steam. In this case a browser extension is used. 2. At one moment music starts playing as background of the show. If the user has activated automatic sound recognition, the system will then record about 30 seconds of the sound with a small visible notification on the extensions browser icon. If the user has not enable automatic recognition, he can still activate the service manually by clicking on the browser icon and activate sound recognition. 3. When the recording is done, the icon indicates that the processing is happening. This is all happening with very subtle changes in the icon, as not to distract the user from their actual activity of watching the series and is not visible at all if the user chooses to watch in full-screen mode. In this 108

111 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service moment the sound information is sent to the relevant API for example Shazam. And the artist and track information in returned. 4. Once the information has been received, the extension will display a small notification about which artist and song it was and the note that it has been added to the users listening history and database. If the user chooses, he can turn these notifications off. 5. After the user finishes watching his TV-series, he is curious about the music and opens the services web app and logs into his account. 6. Upon entering his profile, he will be greeted with a short update on what has happened since his last login. This mainly consists of a list of the music that has been added through outside sources and separately the results of his own music exploration agents which have been working in his absence. 7. As he is curious about the song he just heard in the series, the user focuses on the list of added songs which are chronologically ordered. He can now choose to: a. Rate the song directly, moving between Don t like it and Love it. And might choose to give it an extra tag that highlights some property of the song that the user enjoys. This is to ensure that the user can take note of why he is interested in the song. As it doesn t need to be the case that a user like a whole song but finds one aspect of it of high interest. b. Explore the artists information. He can choose to read about the artists biography and other related information, which can be viewed partly in a standard wiki -like entry and partly in a knowledge graph representation that is specifically geared to show information as relation of this artist and other sources. Such as connected artists- musically and socially and events the artist participated in. 109

112 AI-based Music Discovery Application Design c. Similar to the information view, the user can jump directly to the genre view, where he can focus on this artists position in the overall musical spectrum and see better where the artist is located in comparison to other music the user has previously listened to and which kind of music and genres are in the area between them. d. Within the genre view, but also directly from the overview, the user can release an agent and customize some options like if the user wants the agent to focus on the artists music only or its surrounding area. Also, which agent he wants to release. e. If the user has a service like Spotify connected, he can also choose to listen to this and jump to the other tracks of this artist in the discography overview. 8. In our case, the user chooses first to explore the genre view. He realizes that the artist is actually a couple genres removed from his usual listening preferences. 9. He now activates the marker tools and draws a narrow path between his main listening area to the new artist with a larger circle directly around the artists. 10. The user then clicks on the agent button and chooses his general listening agent. He drag and drops it into the marked area and specifies that he wants the agent to create a playlist with possible songs he might like. 11. As the agent starts his work, the user separately explores the artist s work himself, adding some of their songs to his playlist. At the same time in the background he can observe the agent making his way through the different artists, marking points of interest and what has been added to a new discovery playlist for the user. 110

113 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service 12. As the user doesn t have time right now to listen also to the playlist, he leaves his pc running while going to the kitchen to get some food, ending the current session. 6.8 Interface Design As our service is trying to make the process of using AI and ML techniques more accessible and partly more transparent for the user, one choice for the visual style of the interface was to use transparency in the design and keep the colors light too. The main functionality drives the general layout of the interface, as the map-style visualization favors a large screen real-estate for its use. Therefore, other elements have been designed to float over the map and reduce other input functions to the lower end of the interface. Only the scouting function required stronger user focus and profits from a secondary, standalone design, so that visual clutter is reduced, and the interface shows the relevant functions. Still, as this is a music software where continuous listening is possible, the lower controls would persist. The following shows example screen design with their respective functions. The main user interface section is dominated by the map view - Figure 20. This expresses itself as a cloud-point or landscape of music. Through the map options, the user can activate different properties of the map, for example this would be the genre view- shown in. But also the way the points are arranged could be changed. As ML is very capable in reducing dimensional data, different features could be selected by the user and then fittingly arranged. Examples are clustering based on sound, year, 111

114 AI-based Music Discovery Application Design album, genres etc. To access these different settings, the user have to activate the Map button in the lower right corner in Figure 20. Figure 20 Genre view Figure 21 shows the situation when the user searches for artists over the search bar in the lower left corner. The chosen bands then get listed in the selection field and highlighted on the music map. Additionally, the user starts to see the spread of the related artists. In Figure 22 the user hovers over bands in the area and navigate the map. If he chooses to, once he rests the cursor over a particular band for a moment, a music sample of the band will be played, and more information will be visible. 112

115 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service Figure 21 Search with related artists area Figure 22 Search with related artists area and info sign 113

116 AI-based Music Discovery Application Design Over the option bar in the lower right corner, visible in Figure 23, the user can access different functions- Library, Agents, Markers and Map options. The library gives access to the playlists and saved songs. The Agent functions, as visible in Figure 23 lets the user either reuse an existing agent or create a new one. Figure 23 Agent options After clicking on the option button in the scout selection in Figure 23, the Scout Training View opens - Figure 24. In setting up a Machine Learning mechanic, the whole process takes different stages. Of course, the user will not be in charge of setting up the algorithm itself, but through the current interface, he gains access to two different stages, the training and the evaluation. Furthermore, it has to be said that most likely this sort of training will only encompass a small part of the algorithm models training. As ML algorithms require huge amounts of data which in practicality means at least thousands of data points, the here presented 114

117 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service steps are small parts of an overall training that refines a model, which then is able to adapt to individual users. You can say that the algorithm overall learns a model which is able to learn quickly from users input as discussed before. Figure 24 Scout Training View In this screen, the user is presented with the ML related interface that shows on the left side the existing list of past scouts plus the option to add a new one. In the center the user is prompted to add the training data, which means the reference music that the user wants to find related music to. Here the user can either search single artists/albums/songs or can quickly select his existing playlists to add different works quickly. On the right hand the user has two categories settings and training data. The settings category gives access to different options that could influence the work of the algorithm. The two here presented examples are both concerning the use of broader data User data from other users of the same software as well as the use of this users past 115

118 AI-based Music Discovery Application Design listening history, but other options could further the influence on the algorithm, like a choice of different mathematical algorithm models. Lastly, the interface shows the before added training data. You can see that the training data also contains something that in ML terms refers to as label. This is one of the important aspects in the procedure, as the users can clearly specify not just music that they like, but also ones that they dislike. This mechanism is something that is usually implicated in other software through dislike features, but by implementing this in a broader way, the algorithms should be able to gain a lot of granularity of the users taste profile. In Figure 25, the user triggered the scout option by moving the centrally located switch to the right side. This now prompts the user to choose a starting point for the scout s exploration. If a scout is already active, it would display current results on the right side. The user can choose to figuratively drag the scout to a starting position by clicking release on map, which would close the ML interface and let the user click on the current map view. Alternatively, he can search for a specific artist/album/song to start from. In both ways, the currently active map will be important, as this is the data the scout will choose from, another aspect of how the user engages in interactively influencing the algorithm and exploration. 116

119 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service Figure 25 Scout Information View Once the scout is activated, it will show a representation of the algorithm on the map - Figure 26. This shows the algorithm and which data it currently analyses. Results will be displayed on the right-hand side, where songs are listed for the user together with a recommendation indication in form of a number. This is a mechanic taken from machine learning, as it is used to show a ML algorithms confidence in classifying the result as likeable for the user. Double clicking a song would ideally start playing it, and right clicking gives further options. In general, it would be interesting to explore further options as for example giving estimates on a genre/artist/album/song level and how this would impact the interaction. The benefit being that quicker exploration of more widespread parts of genres would be possible. 117

120 AI-based Music Discovery Application Design Figure 26 Scout Map View Through the marker option, the user can highlight which area he wants to explore and which one he wants to avoid. He can either paint manually or choose genres as a quick way to mark areas, shown in Figure 27. This is an added option for the user to influence the performance of the algorithm. Additionally, the to avoid section can be seen as negative training data for the algorithm, although it might have to weighed lower than specific negative input. 118

121 Chapter 8 Design of a Collaborative AI-Human Music Discovery Service Figure 27 Genre view with marker 6.9 Validation Interviews As a last step for the concept design, the proposed concept was presented to the same interviewees who had taken part in the beginning of the design process. The intention of this step is to validate if the thought process since the first questions is going into a direction that suits these users. As a workable prototype is not in the scope of this study, the interviewees have been given a pdf with the same screens as discussed in chapter 6.8 and accompanying short descriptions of intended features and user interaction of the software. The participants were then asked to answer a short online questionnaire with 20 questions. This questionnaire was segmented into three pages. Page one was asking about their current software use for music and how useful they would rate the software. Page two was similarly asking for their software use and its usefulness for 119