{"id":494,"date":"2020-05-31T15:55:49","date_gmt":"2020-05-31T22:55:49","guid":{"rendered":"https:\/\/blogs.ubc.ca\/taste\/?page_id=494"},"modified":"2020-07-20T22:21:51","modified_gmt":"2020-07-21T05:21:51","slug":"494-2","status":"publish","type":"page","link":"https:\/\/blogs.ubc.ca\/taste\/494-2\/","title":{"rendered":"NIME2020 Poster"},"content":{"rendered":"<table style=\"height: 53px;\" width=\"794\">\n<tbody>\n<tr>\n<td style=\"text-align: center; color: black;\">\n<h2>Inexpensive Colour Tracking to Overcome Performer ID Loss<\/h2>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<table style=\"height: 121px;\" width=\"800\">\n<tbody>\n<tr>\n<td style=\"text-align: center; color: black;\"><strong>Bob Pritchard<\/strong><br \/>\nUBC School of Music<br \/>\n6361 Memorial Road<br \/>\nVancouver, Canada<br \/>\nbob@mail.ubc.ca<\/td>\n<td style=\"text-align: center; color: black;\"><strong>Ian Lavery<\/strong><br \/>\nAlgorhythmic Software Design<br \/>\n308 &#8211; 988 16th Ave W.<br \/>\nVancouver, Canada<br \/>\nian_lavery@hotmail.com<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><iframe loading=\"lazy\" title=\"Cheap Loops - Pritchard\/Fredriksson\/Lavery 2019\" width=\"740\" height=\"416\" src=\"https:\/\/www.youtube.com\/embed\/EpLqE7oQSpA?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<h2 style=\"text-align: left; color: black;\">OVERVIEW<\/h2>\n<table width=\"800\">\n<tbody>\n<tr>\n<td><span style=\"color: #000000;\">Inexpensive single camera motion tracking systems using active infrared (e.g. Kinect, Orbbec) are often used to generate data for use in artistic performance. The Kinect Controlled Artistic Sensing System (KiCASS) is one such environment, developed in 2016 at the University of British Columbia and upgraded in subsequent years.(KiCASS) began as a project between undergraduate Engineering students and the School of Music digital performance ensemble SUBCLASS. It uses a Kinect-for-Windows\u00a9 camera, a high-end Windows laptop, a router, and NuiTrack-based custom written software, now allowing users to select up to 22 tracking points as well as four hand positions on up to six performers. \u00a0While it can track six performers is generally used to track one or two dancers or musicians. It has been used in dozens of studio and public performances and gallery installations, with the generated data\u00a0controlling audio, video, processing, and\/or lighting.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>THE PROBLEMS<\/h2>\n<table width=\"800\">\n<tbody>\n<tr>\n<td>\n<p><a href=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/KiCASSExampleBordered.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-564 alignleft\" src=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/KiCASSExampleBordered-228x300.png\" alt=\"\" width=\"348\" height=\"434\" \/><\/a><\/p>\n<p><span style=\"color: #000000;\"><strong>1. Tracking Loss<\/strong><\/span><\/p>\n<ul>\n<li><span style=\"color: #000000;\">Performer can be occluded by another performer or object, or\u00a0performer will exit and re-enter the tracking area<\/span><\/li>\n<li><span style=\"color: #000000;\">When this occurs system loses track of performer<\/span><\/li>\n<li><span style=\"color: #000000;\">System creates new tracking ID upon reacquisition of performer<\/span><\/li>\n<li><span style=\"color: #000000;\">Original tracking points on performer are lost,\u00a0so no performance data is generated.<\/span><\/li>\n<\/ul>\n<p><span style=\"color: #000000;\"><strong>2. Tracking Swapping<\/strong><\/span><\/p>\n<ul>\n<li><span style=\"color: #000000;\">System will occasionally swap IDs of onscreen performers<\/span><\/li>\n<li><span style=\"color: #000000;\">Performers lose assigned target points, thus unable to control media<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>THE SOLUTION<\/h2>\n<table width=\"800\">\n<tbody>\n<tr>\n<td style=\"color: black;\">\n<p><span style=\"color: #000000;\">To overcome these issues, we developed colour tracking to override system-assigned performer IDs. We generate consistent performer IDs based on costume colour.<\/span><\/p>\n<p><a href=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/RehearsalShotSmall.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-587\" src=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/RehearsalShotSmall.jpg\" alt=\"\" width=\"504\" height=\"378\" srcset=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/RehearsalShotSmall.jpg 504w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/RehearsalShotSmall-300x225.jpg 300w\" sizes=\"auto, (max-width: 504px) 100vw, 504px\" \/><\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Colour Registration of Performers<\/h2>\n<table width=\"800\">\n<tbody>\n<tr>\n<td><a href=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/Selection2GUI.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignleft wp-image-532\" src=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/Selection2GUI-300x285.png\" alt=\"\" width=\"400\" height=\"380\" srcset=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/Selection2GUI-300x285.png 300w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/Selection2GUI.png 633w\" sizes=\"auto, (max-width: 400px) 100vw, 400px\" \/><\/a><\/p>\n<ul>\n<li><span style=\"color: #000000;\">We take a 6-frame analysis of each dancer\u2019s dominant hue, using the quadrilateral created by the joint-tracked shoulders and hip, and NuiTrack\u2019s user mask.<\/span><\/li>\n<li><span style=\"color: #000000;\">Convert the image from RGB to HSV (Hue Saturation Value), using the Open C library Emgu CV<\/span><\/li>\n<li><span style=\"color: #000000;\">Create a histogram of all hue values in the torso region. The colour value that occurs most frequently is selected as the dominant hue of the torso.<\/span><\/li>\n<li><span style=\"color: #000000;\">During registration\u00a0we also select the target points to be tracked on each dancer.<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Data Check<\/h2>\n<table width=\"800\">\n<tbody>\n<tr>\n<td><span style=\"color: #000000;\"><a href=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchColour.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignleft wp-image-749 size-thumbnail\" src=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchColour-150x150.jpg\" alt=\"\" width=\"150\" height=\"150\" srcset=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchColour-150x150.jpg 150w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchColour-144x144.jpg 144w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchColour-50x50.jpg 50w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchColour-125x125.jpg 125w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a>System checks incoming HSV values from each performer every 500 msec.<\/span><\/p>\n<ul>\n<li><span style=\"color: #000000;\">Confidence Value (CV) generated by comparing incoming colour with registered colours<\/span><\/li>\n<li><span style=\"color: #000000;\">CV = 100 \u2013 distance-from-match. Value of 0 means performer has been lost<\/span><\/li>\n<li><span style=\"color: #000000;\">Reacquisition: software determines dominant incoming colour<\/span><\/li>\n<li><span style=\"color: #000000;\">Assigns to a colour through closest distance-from-match<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Optimization<\/h2>\n<table width=\"800\">\n<tbody>\n<tr>\n<td><span style=\"color: #000000;\"><a href=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchOrOptimize.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"size-thumbnail wp-image-750 alignleft\" src=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchOrOptimize-150x150.jpg\" alt=\"\" width=\"150\" height=\"150\" srcset=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchOrOptimize-150x150.jpg 150w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchOrOptimize-144x144.jpg 144w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchOrOptimize-50x50.jpg 50w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/matchOrOptimize-125x125.jpg 125w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a>Every 500 msec each performer&#8217;s colour reanalyzed<\/span><\/p>\n<ul>\n<li><span style=\"color: #000000;\">With three confirmed matches of colour of all performers, system locks matches until CV for any performer falls below certain threshold<\/span><\/li>\n<li><span style=\"color: #000000;\">If CV falls below certain threshold, system runs optimization routines<\/span><\/li>\n<li><span style=\"color: #000000;\">If performer\u2019s colour match changes to shorter distance than current match, colour IDs are swapped between two users identified with those colours<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Performance<\/h2>\n<table width=\"800\">\n<tbody>\n<tr>\n<td><img decoding=\"async\" class=\"alignright\" src=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/CheapLoopsVibrance2.jpg\" alt=\"\" width=\"400\" \/><\/p>\n<ul>\n<li style=\"text-align: left;\"><span style=\"color: #000000;\">Data (X Y Z coordinates of tracked points and CV) sent over wi-fi in OSC format<\/span><\/li>\n<li style=\"text-align: left;\"><span style=\"color: #000000;\">Clients receive data and use Max or Pd to control audio\/video, processing, lighting<\/span><\/li>\n<li style=\"text-align: left;\"><span style=\"color: #000000;\">Tracking area parallelogram has depth of 8 metres and far width of 6 metres<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Outcomes<\/h2>\n<table width=\"800\">\n<tbody>\n<tr>\n<td>\n<ul>\n<li><span style=\"color: #000000;\">Successful tracking four dancers to control amplitude of individual audio tracks<\/span><\/li>\n<li><span style=\"color: #000000;\">Amplitude based on distance of torso from camera<\/span><\/li>\n<li><span style=\"color: #000000;\">Occlusion or exit from tracking area caused associated audio track fade out<\/span><\/li>\n<li><span style=\"color: #000000;\">Re-entry\/acquisition caused associated audio track fade in<\/span><\/li>\n<li><span style=\"color: #000000;\">Issues with shine from theatrical lighting on spandex leotards<\/span><\/li>\n<li><span style=\"color: #000000;\">Problems with shadows being cast on performers<\/span><\/li>\n<li><span style=\"color: #000000;\">Performers\/choreographer enjoyed being able to control audio levels through basic positioning, not interfering with other performance movements<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Future work<\/h2>\n<table style=\"width: 500px;\" width=\"500\">\n<tbody>\n<tr>\n<td style=\"width: 500px;\"><a href=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/DanielleOrange-scaled.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-759 alignleft\" src=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/DanielleOrange-134x300.jpg\" alt=\"\" width=\"147\" height=\"329\" srcset=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/DanielleOrange-134x300.jpg 134w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/DanielleOrange-457x1024.jpg 457w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/DanielleOrange-768x1721.jpg 768w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/DanielleOrange-685x1536.jpg 685w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/DanielleOrange-914x2048.jpg 914w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/DanielleOrange-scaled.jpg 1142w\" sizes=\"auto, (max-width: 147px) 100vw, 147px\" \/><\/a><\/p>\n<ul>\n<li><span style=\"color: #000000;\">Colour registration will include performers moving through space during registration to account for lighting differences<\/span><\/li>\n<li><span style=\"color: #000000;\">Addition of limb tracking and target points to increase control and artistic possibilities<\/span><\/li>\n<li><span style=\"color: #000000;\">Addition of P.I.\u2019s e-textile sensors to increase control opportunities<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Acknowledgements<\/h2>\n<p><span style=\"color: #000000;\">We acknowledge that the University of British Columbia is located on the traditional, ancestral, and unceded territory of the h\u0259n\u0313q\u0313\u0259min\u0313\u0259m\u0313-speaking x\u02b7m\u0259\u03b8k\u02b7\u0259y\u0313\u0259m (Musqueam) people.<\/span><\/p>\n<p><a href=\"https:\/\/blogs.ubc.ca\/taste\/files\/2019\/12\/IMG_1961-scaled.jpeg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-393 size-medium\" src=\"https:\/\/blogs.ubc.ca\/taste\/files\/2019\/12\/IMG_1961-300x225.jpeg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/blogs.ubc.ca\/taste\/files\/2019\/12\/IMG_1961-300x225.jpeg 300w, https:\/\/blogs.ubc.ca\/taste\/files\/2019\/12\/IMG_1961-1024x768.jpeg 1024w, https:\/\/blogs.ubc.ca\/taste\/files\/2019\/12\/IMG_1961-768x576.jpeg 768w, https:\/\/blogs.ubc.ca\/taste\/files\/2019\/12\/IMG_1961-1536x1152.jpeg 1536w, https:\/\/blogs.ubc.ca\/taste\/files\/2019\/12\/IMG_1961-2048x1536.jpeg 2048w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p><a href=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/SSHRC-CRSH_FIP.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-756 size-large alignleft\" src=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/SSHRC-CRSH_FIP-1024x43.jpg\" alt=\"\" width=\"540\" height=\"23\" srcset=\"https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/SSHRC-CRSH_FIP-1024x43.jpg 1024w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/SSHRC-CRSH_FIP-300x13.jpg 300w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/SSHRC-CRSH_FIP-768x32.jpg 768w, https:\/\/blogs.ubc.ca\/taste\/files\/2020\/05\/SSHRC-CRSH_FIP.jpg 1507w\" sizes=\"auto, (max-width: 540px) 100vw, 540px\" \/><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Inexpensive Colour Tracking to Overcome Performer ID Loss Bob Pritchard UBC School of Music 6361 Memorial Road Vancouver, Canada bob@mail.ubc.ca Ian Lavery Algorhythmic Software Design 308 &#8211; 988 16th Ave W. Vancouver, Canada ian_lavery@hotmail.com OVERVIEW Inexpensive single camera motion tracking systems using active infrared (e.g. Kinect, Orbbec) are often used to generate data for use [&hellip;]<\/p>\n","protected":false},"author":23393,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-494","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/blogs.ubc.ca\/taste\/wp-json\/wp\/v2\/pages\/494","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.ubc.ca\/taste\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/blogs.ubc.ca\/taste\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.ubc.ca\/taste\/wp-json\/wp\/v2\/users\/23393"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.ubc.ca\/taste\/wp-json\/wp\/v2\/comments?post=494"}],"version-history":[{"count":233,"href":"https:\/\/blogs.ubc.ca\/taste\/wp-json\/wp\/v2\/pages\/494\/revisions"}],"predecessor-version":[{"id":763,"href":"https:\/\/blogs.ubc.ca\/taste\/wp-json\/wp\/v2\/pages\/494\/revisions\/763"}],"wp:attachment":[{"href":"https:\/\/blogs.ubc.ca\/taste\/wp-json\/wp\/v2\/media?parent=494"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}