Representation of Musical Information

Representation of Musical Information

Other Topics in Music Organization/Representation Donald Byrd School of Informatics Indiana University Updated 29 April 2006 Copyright 2003-06, Donald Byrd 1 Classification: Surgeon Generals Warning Classification (ordinary hierarchic) is dangerous Almost everything in the real world is messy Absolute correlations between characteristics are rare Example: some mammals lay eggs; some are naked

Example: musical instruments (piano as percussion, etc.) Nearly always, all you can say is an X has characteristic A, and usually also B, C, D Leads to: People who know better claiming absolute correlations Arguments among experts over which characteristic is most fundamental Don changing his mind 30 Jan. 06 2 Comparison of Music-IR Task Classifications Typke, Rainer, Wiering, Frans, & Veltkamp, Remco C. (2005). A Survey of Music Information Retrieval Systems Overview of 17 systems for content-based retrieval of music in both audio & symbolic forms Includes map of systems showing tasks & users for which each is most appropriate

Horizontal axis (tasks) has similar idea to my Similarity Scale for Content-Based Music IR Main difference: Typke et al have artist Doesnt fit hierarchy, but useful--and dangerous! 4 April 06 3 Music Recommender Systems (1) Guest speaker: Justin Donaldson PhD student, IU Computer Science Intern, MusicStrands Pandoras approach Classification by experts with controlled vocabulary Music genome MusicStrands approach Co-occurrence, network analysis, with limited guidance by expert

Examples: FOAFing the Music,, MusicIP Mixer, Musicmatch Jukebox, MusicStrands, Pandora 5 April 06 4 Music Recommender Systems (2) All(?) existing systems try to find music similar to what you give them Instead, do the opposite Tim Crawford to Don (2004): I don't want to find more music like what I already know, I want music as different as possible from it! Jeremy Pickens' example: Eigenradio NOT a good example! How to automate? Genre classification? 5 April 06

5 Music Recommender Systems (3) Systems: FOAFing the Music,, MusicIP Mixer, Musicmatch Jukebox, MusicStrands, Pandora Pandoras music genome idea Assumes all music based on small number of genes Content-based Requires annotation by human experts Conventional collaborative approach(?) Others? 7 April 06 6 Maps, Visualizations, & Metrics (1)

Example 1: Justin Donaldsons 3-D visualization Map requires metric (similarity measure) => positions in n-dimensional space n can be huge, except for visualization Example 2: Pampalk et al, Exploring Music Collections by Browsing Different Views (ISMIR 2003, CMJ 2004) Organization by spectrum, periodicity, metadata Uses self-organizing maps (SOMs) SOMs can focus on audio analysis and/or metadata Maps of same collection aligned => can move from one view to another 10 April 06 7

Maps, Visualizations, & Metrics (2) With good similarity measure, easy to find similar stuff or different stuff! Automatically (searching, filtering) Do-it-yourself (browsing) Whats a good similarity measure for music? Usual interpretation: what are good features to use? Cf. Eigenradio: has objective features, but need subjective Cf. Pampalk comment on main difficulty 10 April 06 8 Digital Music Libraries

iTunes: no. 1 commercial system Popular & simple but not always easy to find music with Not a real music library: does very little Variations2: research project => production system 10 April 06 9 What is a Digital Library? Not just library with computers & on-line catalog! DL as collection/information system collection of information that is both digitized and organized -- Michael Lesk, NSF networked collections of digital text, documents, images, sounds, scientific data, and software -- PITAC report DL as organization organization that provides resources to select, structure, offer intellectual access to, interpret, distribute, preserve integrity of,

and ensure persistence over time of collections of digital works... -- Digital Library Federation Elephant in the Room for all DLs: persistence over time = preservation 13 April 06 10 What is a Digital Music Library? Music has many special needs Content formats Need audio, scores; want video, maybe MIDI, etc. Search capabilities for content and metadata Intellectual Property Rights (IPR) => access control important Traditional library catalogs dont handle music

well One reason: lack of music-specific metadata 13 April 06 11 Variations and Variations2 Digital library of music sound recordings & scores Original concept 1990, online since 1996 Variations2 started as pure research project Now production system; replaced Variations in 2005 Accessible by all in Music Library; other locations restricted for IPR reasons Used daily by large student population Currently: 11,500 titles, 15,000 hours of audio

Over 6 TB uncompressed, 1.6 TB compressed (MP3, AAC) Opera, songs, instrumental music, jazz, rock, world music, etc. 13 April 06 12 Some Metadata and Digital Library Buzzwords MARC: metadata standard for library catalogs From the Library of Congress Old (1970s): fixed format, etc.; bibliographic Standard for maintaining & exchanging bibliographic information Simple relationships, elaborate details Dublin Core (DC): general-purpose metadata standard

From Dublin Core Metadata Initiative (DCMI) New (1990s): XML, etc.; metadata Simple, general, extensible Terminology: Open Archives Initiative (OAI): metadata consumer FRBR: metadata standard for library catalogs From IFLA, with support from Library of Congress, etc. New (>2000) Complex relationships, elaborate details 18 April; rev. 25 April 13 Functional Requirements for Bibliographic Records (FRBR) Represents much more complex relationships than MARC MARC records refer explicitly to subject headings (LCSH), URLs and implicitly (via uniform names & titles) to other MARC records but not consistently!

FRBR (like Variations2) records always refer to each other FRBR Entities Group 1: Products of intellectual & artistic endeavor Group 2: Those responsible for the intellectual & artistic content Group 3: Subjects of works Much of following from by Barbara Tillett (2002), The FRBR Model (Functional Requirements for Bibliographic Records) 25 April 14 FRBR Entities Group 1: Products of intellectual & artistic endeavor 1. Work (completely abstract) 2. Expression 3. Manifestation 4. Item (completely concrete: you can touch one) Almost heirarchic; almost since works can include other works

Group 2: Those responsible for the intellectual & artistic content Person Corporate body Group 3: Subjects of works Groups 1 & 2 plus Concept Object Event Place 25 April 15

Relationships of Group 1 Entities: Example w1 J.S. Bachs Goldberg Variations e1 Performance by Glenn Gould in 1981 m1 Recording released on 33-1/3 rpm sound disc in 1982 by CBS Records i1a, 1b, 1c Copies of that 33-1/3 rpm disc acquired in 1984-87 by the Cook Music Library m2 Recording re-released on compact disc in 1993 by Sony i2a, i2b Copies of that compact disc acquired in 1996 by the Cook Music Library m3 Digitization of the Sony re-released as MP3 in 2000 25 April 16 Relationships of Group 1 Entities (1)

Work Expression is realized through Intellectual/ artistic content Physical recording of content Manifestation Item 25 April is embodied in is exemplified by

17 Relationships of Group 1 Entities (2) Work is realized through Expression is embodied in recursive one Manifestation is exemplified by many Item 25 April 18

FRBR vs. Variations2 Data Models FRBR Group 1 Work Expression Manifestation Item Group 2 Person Corporate body (any named organization?) Variations2

Work Instantiation Media object Contributor (person or organization) Container Items in blue are close, though not exact, equivalents. 25 April; rev. 26 April 19 Elephant in the Room for Music DLs: Getting Catalog Information into FRBR or Variations2 2005 MLA discussion Cataloging to current standards (MARC) is very expensive FRBR and Variations2 both much more demanding Michael Lesk/NSF: didnt like funding metadata projects because they always said every other project should be more expensive! Libraries seem to be moving to FRBR anyway

Idea 1: collaborative cataloging (ala OCLC) Idea 2: take advantage of existing cataloging Variations2: simple Import MARC feature VTLS: convert MARC => FRBR is much more ambitious Good ideas, but probably not enough Idea 3: user-contributed metadata? 13 April 06 20 Variations2 Hands-on Possibilities

Search using Variations2 search window Search using IUCAT External (WWW or other) access via reserve lists, etc. Create playlist Add bookmarks Create listening drill from playlist Export to make a Web page Use Opus window Use Timeliner 10 April 06 21 Variations2 Data Types Work is realized through Instantiation (recording or score) is embodied in Container (CD, LP, edition of scores, etc.) Contributor (person or organization)

Contributor to work: composer, lyricist, etc. Contributor to instantiation: performer, conductor, engineer, etc. 10 April 06 22 Works & Work Relationships Work concept is new to Variations2/FRBR Much more important to organize music than (e.g.) books Language of title says very little about content Important relationships: song & album, aria & opera, etc. Work relationships can be very complex Part/whole Arrangement Version (improvisation, etc.)

12 April 06 23 Style Genres & Genre Classifications Genre Classifications are a mess No consistency between classifications All-Music Guide: <=4 levels: 2 top-level (pop/classical), 34 second-level ca. 23 main genres 47 genres, flat ID3 tags (in MP3's): 80 genres, flat; WinAmp version: 126 genres, flat Ishkur's EM Guide ("Electronic music" only): <=3 levels: 7 toplevel iTunes: 37 genres, flat MIREX 2005 : 9, 38 leaves 10 April 06 24

Style Genres & Genre Classifications No wonder: what makes a musical style is very subtle! In many cases, "correct" genre can't be determined without knowledge of the lyrics, even understanding or even intent of creators Dave Datta (2005): automatic genre-classification programs are finding something, & probably useful, but may not be genres as people understand them Turntablism is a separate genre--or is it? If it's done "mildly", what you'd hear is mostly the genre of the underlying music! 13 April 06 25 Transcription of Polyphonic Audio Cf. OMRAS experiments: an important research problem Why is it so difficult? Guest speaker: Ian Knopke, IU fellow in music informatics

14 April 06 26 Music Plus One (1) Chris Raphaels accompaniment system Goals of ``Music Plus One'' are similar to ``Music Minus One'' (MMO) But, with MMO, soloist must follow accompaniment Goals: program must: Respond in real time to soloist's tempo changes & expression Learn from past performances so it assimilates soloist's interpretation in future Bring sense of musicality to performance Components: Listen and Play 14 April 06 27 Music Plus One (2)

Listen As soloist plays, signal analyzed to determine what notes have played & exactly when Greatly simplified because computer knows score (MIDI file) But must be robust to inaccuracies & embellishments by soloist while maintaining accuracy in matching signal to soloist's part Uses Hidden Markov Model (HMM) Play: fuse knowledge sources Output of Listen Score (notes, rhythms, etc.) To improve over successive rehearsals, use collection of past solo performances by soloist Performances of the accompaniment by a live player 14 April 06 28 Accompaniment System Variations

Music written for system (beyond human capabilities) Erase the soloist => greatly enlarge repertoire Detecting beat in audio (alignment with score)... Much easier than detecting without the score (e.g., Digital Performer) For use by Variations2? Example: Sacrificial Dance of Le Sacre du Printemps 20 April 06 29 Music as Different as Possible Results are interesting Two teams used Cages 4 33 All lists good, none great

Not much world music! Team A: Ives The Unanswered Question Team B: Prefuse 73s B2 Living Life Team C: Japanese gagaku Keibairaku No Kyu (Taishiki-Cho) Team D: Balinese monkey chants Electronic/computer music can be much more extreme Xenakiss Bohor: few definite-pitched sounds Dodges In Celebration: wild synthesized voice You sit in a chair, touched by nothing, feeling the old self Consider language + basic parameters of sound Pitch, duration/rhythm, dynamics, timbre 17 April 06 30

Expectation and Perception with Sponges, Dinosaurs, and Music Sponges Contamination of kitchen surfaces before & after cleaning with sponge surprised researchers Dinosaurs 1922 audience fooled by test reel for The Lost World Music Dons experience with Kurzweil flute sound Hammond organ Model A compared to pipe organ (ca. 1940) In blind test, experts & students couldnt tell them apart 16 April 06 31 Science, Scholarship, and Critical Thinking

Good research is very hard Electronic Musician article on analog summing The issue isnt just science D. Huron on what he learned about music scholarship It isnt just scholarship 1922 audience fooled by test reel for The Lost World Its critical thinking ALWAYS evaluate information sources ALWAYS consider biases, including your own Darwins attitude about his biases Most people would rather die than think 17 April 06 32 Intellectual Property Rights (IPR) (1) IPR is huge problem for music IT, including IR, both

research & use No one knows the answers! Different in different countries! For music, U.S. copyright is complex bundle of rights mechanical right: use in commercial recordings, ROMs, online delivery for private use synchronization right: use in audio/visual works (movies, TV, etc.) More complex than for text works because performing art U.S. Constitution: balance rights of creators and public To achieve these conflicting goals and serve the public interest requires a delicate balance between the exclusive rights of authors and the long-term needs of a knowledgeable society. Mary Levering, U.S. Copyright Office After some time, work enters Public Domain 26 April 06 33 Intellectual Property Rights (IPR) (2)

Law supposed to balance rights of creators & public, but Time till Public Domain getting longer & longer Joke: When will old Disney movies be Public Domain? Sonny Bono Copyright Extension Act: not till 70 years after death! Digital Millenium Copyright Act (DMCA) restricts owners rights Rep. Smiths bill (in Congress soon?) even worse Fair Use: U.S. limit on exclusive rights of copyright owners Traditionally used for excerpts for reviews, etc. Not well-defined. Four tests: 1. 2. 3. 4.

Purpose and character of use, including if commercial or nonprofit Nature of copyrighted work Amount and substantiality of portion used relative to work as a whole Effect of use on potential market for or value of copyrighted work Law also has educational exemptions 26 April 06 34 Intellectual Property Rights (IPR) (3) NB: Im not a lawyer! IPR in practice sued & shut down Peer-to-Peer networks: Napster, Gnutella, FreeNet Church choir director arranged work, did free performance; donated to publisher => sued Example: Student wants to quote brief excerpts from Beethoven piano sonatas in class paper, in notation Do they need permission from owner?

Beethoven dead for more than 70 years => in Public Domain but not all editions Still, dont need permission: Fair Use applies For recording, probably not P.D., but Fair Use applies 26 March, rev. 15 April 35 Music IR as Music Understanding Dannenberg (ISMIR 2001 invited paper) argued central problem of music IR is music understanding also basis for much of computer music (composition & sound synthesis) and music perception and cognition A key problem in many fields is the understanding and application of human musical thought and processing

Don: No understanding yet => sidestep intractable problems! Cf. how people find information vs. how computers find information 14 April 06 36 Detecting Beats/Tempo in Audio without a score (1) Related tasks: tempo detection & beat detection/slicing What can you do with them? Create loops Change tempo radically with no artifacts Ask Will Pierce State of the art in commercial products Digital Performer Beat Detection Engine Employing sophisticated transient detection

technology Likely to work only with very simple texture 28 April 06 37 Detecting Beats/Tempo in Audio without a score (2) What else can you do? More advanced stuff: Change swing feel to straight 8ths (Digital Performer) State of the art in research systems MIREX 2005 Audio Tempo Extraction contest Looking for notated & perceived tempo and phase (= upbeat)

Music w/stable tempo, wide variety of styles, many non-Western Texture? With beat slicing and audio similarity: Violate IPR laws (Scrambled Hackz!) Interview:,70664-0.html Video: 28 April 06 38 Intellectual Property Rights in The Real World NB: Im not a lawyer! A common way for people to decide whats OK Consider ethics: any problem? Consider practical effects: any problem? If no and no, go ahead Example: Member of this class wants to share copyrighted music with others in the class Ethics: it depends

Practical effects: ordinarily none Thorny issue: at what point is sampling a problem? Deeper, thornier issue: does IPR make sense? Joey Morwick: maybe not Ian Clarke (Freenet), Sven Koenig (Scrambled Hackz), promoters of XOR circumvention: absolutely not! 27 April 06 39 Conclusion; Thank You Please, please think for yourself ALWAYS evaluate information sources ALWAYS consider biases, including your own Schoenberg: This book I have learned from my students Don Byrd: This course I have learned from my students Ive learned a lot about music and technology

Im available! 28 April 06 40

Recently Viewed Presentations

  • Title


    Payless had to change their entire marketing mix. "Place" was altered by fixing up their stores and "Product" now included more designer brands. Students are probably familiar with Payless but there is a web link on this page in case...
  • What does Augustine say about Evil??

    What does Augustine say about Evil??

    WHAT DOES AUGUSTINE SAY ABOUT EVIL? ALL: will know Augustine's Theodicy MOST: will be able to offer criticisms to Augustine's Theodicy SOME: will evaluate whether Augustine's Theodicy solves the problem of evil Fill in the Blanks: The Problem of Evil...
  • You Light Up My Life

    You Light Up My Life

    Speciation Chapter 18
  • Nessun titolo diapositiva

    Nessun titolo diapositiva

    More performant and stable than a single optimized tree. Boosting achieved by iteratively reweighting training sample according to classifier output in previous iteration. Reweight events using previous iteration's classifier result. Build and optimize a new tree with reweighted events ....
  • Welcome to Language Arts!!! -

    Welcome to Language Arts!!! -

    Welcome to Language Arts!!! Mr. Chase Rains. Royal Spring Middle School ... To teach you the content of the English/Language Arts. Our Classroom Rules: ... The reason for this Late Work Policy is because of the designated homework policy previously...
  • Main Title of Presentation Goes Here

    Main Title of Presentation Goes Here

    GP, part time Consultant in mental health trust, Senior lecturer in general practice, and LDN AD. Tavistock and Portman NHS Foundation Trust. Otolaryngology. MTR0115. Consultant Psychiatrist. ... KSS Deanery Trainee Number but is LD trainee.have confirmation from Med Workforce. (15/04/2011)...
  • Syndrome de sjogren Mirieme GHAZI CRRF introduction Le

    Syndrome de sjogren Mirieme GHAZI CRRF introduction Le

    Arial Garamond Times New Roman Verdana Wingdings Wingdings 3 Comic Sans MS Niveau Syndrome de sjogren introduction Historique Épidémiologie Les données physiopathologiques Diapositive 6 Les facteurs génétiques les facteurs environnementaux La cellule épithéliale Dysfonctionnement de la synapse neuroexocrine BAFF (B-cell-activating...
  • Performance Evaluation Systems  Part 1- Fitness Reports  Part

    Performance Evaluation Systems Part 1- Fitness Reports Part

    (a) Take action to resolve inconsistencies and disagreements when the MRO's attached statement disagrees with the RO as to matters of fact. (b) Take action to determine the validity of the new information, when the MRO's statement adds new information...