CINXE.COM

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>LKML: Ingo Molnar: [patch -mm] scheduler cache hot autodetect, print less</title><link href="/css/message.css" rel="stylesheet" type="text/css" /><link href="/css/wrap.css" rel="alternate stylesheet" type="text/css" title="wrap" /><link href="/css/nowrap.css" rel="stylesheet" type="text/css" title="nowrap" /><link href="/favicon.ico" rel="shortcut icon" /><script src="/js/simple-calendar.js" type="text/javascript"></script><script src="/js/styleswitcher.js" type="text/javascript"></script><link rel="alternate" type="application/rss+xml" title="lkml.org : last 100 messages" href="/rss.php" /><link rel="alternate" type="application/rss+xml" title="lkml.org : last messages by Ingo Molnar" href="/groupie.php?aid=18" /><script> var _paq = window._paq = window._paq || []; /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ _paq.push(["setDoNotTrack", true]); _paq.push(["disableCookies"]); _paq.push(['trackPageView']); _paq.push(['enableLinkTracking']); (function() { var u="//m.lkml.org/"; _paq.push(['setTrackerUrl', u+'matomo.php']); _paq.push(['setSiteId', '1']); var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); })(); </script></head><body onload="es.jasper.simpleCalendar.init();" itemscope="itemscope" itemtype="http://schema.org/BlogPosting"><table border="0" cellpadding="0" cellspacing="0"><tr><td width="180" align="center"><a href="/"><img style="border:0;width:135px;height:32px" src="/images/toprowlk.gif" alt="lkml.org" /></a></td><td width="32">聽</td><td class="nb"><div><a class="nb" href="/lkml"> [lkml]</a> 聽 <a class="nb" href="/lkml/2005"> [2005]</a> 聽 <a class="nb" href="/lkml/2005/12"> [Dec]</a> 聽 <a class="nb" href="/lkml/2005/12/10"> [10]</a> 聽 <a class="nb" href="/lkml/last100"> [last100]</a> 聽 <a href="/rss.php"><img src="/images/rss-or.gif" border="0" alt="RSS Feed" /></a></div><div>Views: <a href="#" class="nowrap" onclick="setActiveStyleSheet('wrap');return false;">[wrap]</a><a href="#" class="wrap" onclick="setActiveStyleSheet('nowrap');return false;">[no wrap]</a> 聽 <a class="nb" href="/lkml/mheaders/2005/12/10/58" onclick="this.href='/lkml/headers'+'/2005/12/10/58';">[headers]</a>聽 <a href="/lkml/bounce/2005/12/10/58">[forward]</a>聽 </div></td><td width="32">聽</td></tr><tr><td valign="top"><div class="es-jasper-simpleCalendar" baseurl="/lkml/"></div><div class="threadlist">Messages in this thread</div><ul class="threadlist"><li class="root"><a href="/lkml/2005/12/9/202">First message in thread</a></li><li><a href="/lkml/2005/12/9/202"> hawkes@sgi ...</a><ul><li><a href="/lkml/2005/12/9/255">Paul Jackson</a></li><li class="origin"><a href="">Ingo Molnar</a></li><li><a href="/lkml/2005/12/10/60">Ingo Molnar</a><ul><li><a href="/lkml/2005/12/12/192">"John Hawkes"</a><ul><li><a href="/lkml/2005/12/13/26">Ingo Molnar</a></li></ul></li></ul></li></ul></li></ul><div class="threadlist">Patch in this message</div><ul class="threadlist"><li><a href="/lkml/diff/2005/12/10/58/1">Get diff 1</a></li></ul></td><td width="32" rowspan="2" class="c" valign="top"><img src="/images/icornerl.gif" width="32" height="32" alt="/" /></td><td class="c" rowspan="2" valign="top" style="padding-top: 1em"><table><tr><td><table><tr><td class="lp">Date</td><td class="rp" itemprop="datePublished">Sat, 10 Dec 2005 13:02:32 +0100</td></tr><tr><td class="lp">From</td><td class="rp" itemprop="author">Ingo Molnar <></td></tr><tr><td class="lp">Subject</td><td class="rp" itemprop="name">[patch -mm] scheduler cache hot autodetect, print less</td></tr></table></td><td></td></tr></table><pre itemprop="articleBody"> * hawkes@sgi.com <hawkes@sgi.com> wrote: > The essence of the bug is that the code and data in the CPU Scheduler > that perform the cost calculation are declared as "__initdata" or > "__init" or "__devinit", even though a runtime declaration of a > cpu-exclusive cpuset will invoke build_sched_domains() to rebuilt the > sched domains, and that calls the now-released > calibrate_migration_costs(), et al. The attached patch changes the > declarations of the code and data to make them persistent. ok. Acked-by: Ingo Molnar <mingo@elte.hu> > (1) calibrate_migration_costs() printk's the matrix at boot time, and > for large NR_CPUS values this can be voluminous. We might consider > changing the naked printk()'s into printk(KERN_DEBUG ...) to hide > them to everyone but a sysadmin or developer who has a need to > view the values. yeah, agreed. > (2) calibrate_migration_costs() printk's the matrix for every call to > build_sched_domains(), which is called twice for each declaration > of a cpu-exclusive cpuset. The syslog output is voluminous for > large NR_CPUS values. It is unclear to me how interesting this > output is after the initial display at boot time. Various Job > Manager software will actively create and destroy cpu-exclusive > cpusets, and that will flood the syslog with matrix output. ok. I've changed the code to be less verbose: only the distance function is printed, not the full matrix. I have also made some other printks dependent on migration_debug (which is off by default). The boot-time output is now a minimalistic: migration_cost=269,777 which can also be pasted into the kernel's boot command line to turn off calibration on the next bootup. Patch attached below, goes to the tail of the current scheduler queue in -mm. [i'll reply to your other points in the next mail.] Ingo ---- be less verbose in the migration-cost printout. Signed-off-by: Ingo Molnar <mingo@elte.hu> kernel/sched.c | 52 +++++++++++++--------------------------------------- 1 files changed, 13 insertions(+), 39 deletions(-) Index: linux-mm.q/kernel/sched.c =================================================================== --- linux-mm.q.orig/kernel/sched.c +++ linux-mm.q/kernel/sched.c @@ -5485,10 +5485,8 @@ static unsigned long long measure_migrat void calibrate_migration_costs(void) { int cpu1 = -1, cpu2 = -1, cpu, orig_cpu = raw_smp_processor_id(); + unsigned long j0, j1, distance, max_distance = 0; struct sched_domain *sd; - unsigned long distance, max_distance = 0; - unsigned long long cost; - unsigned long j0, j1; j0 = jiffies; @@ -5502,14 +5500,11 @@ void calibrate_migration_costs(void) distance = domain_distance(cpu1, cpu2); max_distance = max(max_distance, distance); /* - * Do we have the result cached already? + * No result cached yet? */ - if (migration_cost[distance] != -1LL) - cost = migration_cost[distance]; - else { - cost = measure_migration_cost(cpu1, cpu2); - migration_cost[distance] = cost; - } + if (migration_cost[distance] == -1LL) + migration_cost[distance] = + measure_migration_cost(cpu1, cpu2); } } /* @@ -5526,8 +5521,8 @@ void calibrate_migration_costs(void) /* * Print the matrix: */ - printk("---------------------\n"); - printk("| migration cost matrix (max_cache_size: %d, cpu: %d MHz):\n", + if (migration_debug) + printk("migration: max_cache_size: %d, cpu: %d MHz:\n", max_cache_size, #ifdef CONFIG_X86 cpu_khz/1000 @@ -5535,37 +5530,16 @@ void calibrate_migration_costs(void) -1 #endif ); - printk("---------------------\n"); - printk(" "); - for_each_online_cpu(cpu1) - printk(" [%02d]", cpu1); - printk("\n"); - for_each_online_cpu(cpu1) { - printk("[%02d]: ", cpu1); - for_each_online_cpu(cpu2) { - if (cpu1 == cpu2) { - printk(" - "); - continue; - } - distance = domain_distance(cpu1, cpu2); - max_distance = max(max_distance, distance); - cost = migration_cost[distance]; - printk(" %2ld.%ld(%ld)", (long)cost / 1000000, - ((long)cost / 100000) % 10, distance); - } - printk("\n"); - } - printk("--------------------------------\n"); - printk("| cacheflush times [%ld]:", max_distance+1); + printk("migration_cost="); for (distance = 0; distance <= max_distance; distance++) { - cost = migration_cost[distance]; - printk(" %ld.%ld (%Ld)", (long)cost / 1000000, - ((long)cost / 100000) % 10, cost); + if (distance) + printk(","); + printk("%ld", (long)migration_cost[distance] / 1000); } printk("\n"); j1 = jiffies; - printk("| calibration delay: %ld seconds\n", (j1-j0)/HZ); - printk("--------------------------------\n"); + if (migration_debug) + printk("migration: %ld seconds\n", (j1-j0)/HZ); /* * Move back to the original CPU. NUMA-Q gets confused - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at <a href="http://vger.kernel.org/majordomo-info.html">http://vger.kernel.org/majordomo-info.html</a> Please read the FAQ at <a href="http://www.tux.org/lkml/">http://www.tux.org/lkml/</a> </pre></td><td width="32" rowspan="2" class="c" valign="top"><img src="/images/icornerr.gif" width="32" height="32" alt="\" /></td></tr><tr><td align="right" valign="bottom"> 聽 </td></tr><tr><td align="right" valign="bottom">聽</td><td class="c" valign="bottom" style="padding-bottom: 0px"><img src="/images/bcornerl.gif" width="32" height="32" alt="\" /></td><td class="c">聽</td><td class="c" valign="bottom" style="padding-bottom: 0px"><img src="/images/bcornerr.gif" width="32" height="32" alt="/" /></td></tr><tr><td align="right" valign="top" colspan="2"> 聽 </td><td class="lm">Last update: 2005-12-10 13:06 聽聽 [from the cache] 漏2003-2020 <a href="http://blog.jasper.es/">Jasper Spaans</a>|hosted at <a href="https://www.digitalocean.com/?refcode=9a8e99d24cf9">Digital Ocean</a> and my Meterkast|<a href="http://blog.jasper.es/categories.html#lkml-ref">Read the blog</a></td><td>聽</td></tr></table><script language="javascript" src="/js/styleswitcher.js" type="text/javascript"></script></body></html>