CINXE.COM
LLVM updates and bazel cache - Build - Build with Google AI
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>LLVM updates and bazel cache - Build - Build with Google AI</title> <meta name="description" content="As we are updating LLVM two times at day I’ve tried to query bazel with: bazel aquery &quot;rdeps(//tensorflow:*,//third_party/llvm:*)&quot; --include_aspects 2&gt;/dev/null | grep Compiling | wc -l I am not a bazel ninja so proba&hellip;"> <meta name="generator" content="Discourse 3.4.0.beta3-dev - https://github.com/discourse/discourse version d880db3b7b748901366d0c94f58dada9096f967f"> <link rel="icon" type="image/png" href="https://canada1.discourse-cdn.com/googleaitrial/optimized/2X/8/8bfe64fc593eb7baba8df149ebba4457c16ca1f9_2_32x32.png"> <link rel="apple-touch-icon" type="image/png" href="https://canada1.discourse-cdn.com/googleaitrial/optimized/2X/8/8bfe64fc593eb7baba8df149ebba4457c16ca1f9_2_180x180.png"> <meta name="theme-color" media="all" content="#1f1f1f"> <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0, user-scalable=yes, viewport-fit=cover"> <link rel="canonical" href="https://discuss.ai.google.dev/t/llvm-updates-and-bazel-cache/31793" /> <link rel="search" type="application/opensearchdescription+xml" href="https://discuss.ai.google.dev/opensearch.xml" title="Build with Google AI Search"> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/color_definitions_google-ai-for-developers_8_2_9664c403088a1d589ded99d7cdbbd8f338343719.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" class="light-scheme"/> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/desktop_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="desktop" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/checklist_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="checklist" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-ai_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-ai" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-akismet_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-akismet" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-cakeday_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-cakeday" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-data-explorer_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-data-explorer" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-details_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-details" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-lazy-videos_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-lazy-videos" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-local-dates_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-local-dates" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-narrative-bot_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-narrative-bot" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-policy_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-policy" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-presence_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-presence" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-reactions_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-reactions" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-solved_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-solved" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-templates_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-templates" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-topic-voting_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-topic-voting" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/footnote_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="footnote" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/hosted-site_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="hosted-site" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/poll_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="poll" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/spoiler-alert_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="spoiler-alert" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-ai_desktop_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-ai_desktop" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-reactions_desktop_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-reactions_desktop" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/discourse-topic-voting_desktop_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="discourse-topic-voting_desktop" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/poll_desktop_5a2434b361468bc0f0245e49d5dee554eebac4e3.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="poll_desktop" /> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/desktop_theme_4_7896a10e11b78b3457576dae80aa6a4f54f8c407.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="4" data-theme-name="discourse header search"/> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/desktop_theme_3_8fbb151033f3941d27908950c0e87f8fb39a38a6.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="3" data-theme-name="discourse-material-icons"/> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/desktop_theme_9_f0575d150ab7e99c44052a5015695044e12c5b03.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="9" data-theme-name="global notice custom css"/> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/desktop_theme_8_afb04a60a9c25410f7b14d8fa0f3ae4ed9169b51.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="8" data-theme-name="temp nav fixes"/> <link href="https://yyz1.discourse-cdn.com/googleaitrial/stylesheets/desktop_theme_2_6b14ef109bd30716a9d1b973997ba1349dda629b.css?__ws=discuss.ai.google.dev" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="2" data-theme-name="google ai"/> <link href="https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:opsz,wght,FILL,GRAD@20..48,100..400,0..1,0&family=Roboto+Mono:ital,wght@0,100..700;1,100..700&family=Roboto:ital,wght@0,100;0,300;0,400;0,500;0,700;0,900;1,100;1,300;1,400;1,500;1,700;1,900&display=swap" rel="stylesheet"> <link rel="alternate nofollow" type="application/rss+xml" title="RSS feed of 'LLVM updates and bazel cache'" href="https://discuss.ai.google.dev/t/llvm-updates-and-bazel-cache/31793.rss" /> <meta property="og:site_name" content="Build with Google AI" /> <meta property="og:type" content="website" /> <meta name="twitter:card" content="summary" /> <meta name="twitter:image" content="https://canada1.discourse-cdn.com/googleaitrial/original/2X/8/8bfe64fc593eb7baba8df149ebba4457c16ca1f9.png" /> <meta property="og:image" content="https://canada1.discourse-cdn.com/googleaitrial/original/2X/8/8bfe64fc593eb7baba8df149ebba4457c16ca1f9.png" /> <meta property="og:url" content="https://discuss.ai.google.dev/t/llvm-updates-and-bazel-cache/31793" /> <meta name="twitter:url" content="https://discuss.ai.google.dev/t/llvm-updates-and-bazel-cache/31793" /> <meta property="og:title" content="LLVM updates and bazel cache" /> <meta name="twitter:title" content="LLVM updates and bazel cache" /> <meta property="og:description" content="As we are updating LLVM two times at day I’ve tried to query bazel with: bazel aquery "rdeps(//tensorflow:*,//third_party/llvm:*)" --include_aspects 2>/dev/null | grep Compiling | wc -l I am not a bazel ninja so probably the query could be wrong or improved but I currently see 9938 files on master (CPU only). What is the effect of this bi-daily rolling update on the average community contributor compiling workflow/environment and his bazel cache?" /> <meta name="twitter:description" content="As we are updating LLVM two times at day I’ve tried to query bazel with: bazel aquery "rdeps(//tensorflow:*,//third_party/llvm:*)" --include_aspects 2>/dev/null | grep Compiling | wc -l I am not a bazel ninja so probably the query could be wrong or improved but I currently see 9938 files on master (CPU only). What is the effect of this bi-daily rolling update on the average community contributor compiling workflow/environment and his bazel cache?" /> <meta property="og:article:section" content="TensorFlow" /> <meta property="og:article:section:color" content="f66f00" /> <meta property="og:article:section" content="Special Interest Groups" /> <meta property="og:article:section:color" content="8e5de8" /> <meta property="og:article:section" content="Build" /> <meta property="og:article:section:color" content="AF93E4" /> <meta property="og:article:tag" content="sig_mlir" /> <meta property="og:article:tag" content="sig_build" /> <meta property="og:article:tag" content="contributing" /> <meta property="og:article:tag" content="community" /> <meta property="og:article:tag" content="help_dev" /> <meta name="twitter:label1" value="Reading time" /> <meta name="twitter:data1" value="8 mins 🕑" /> <meta name="twitter:label2" value="Likes" /> <meta name="twitter:data2" value="10 ❤" /> <meta property="article:published_time" content="2021-06-16T22:31:47+00:00" /> <meta property="og:ignore_canonical" content="true" /> <link rel="next" href="/t/llvm-updates-and-bazel-cache/31793?page=2"> </head> <body class="crawler browser-update"> <header> <a href="/"> Build with Google AI </a> </header> <div id="main-outlet" class="wrap" role="main"> <div id="topic-title"> <h1> <a href="/t/llvm-updates-and-bazel-cache/31793">LLVM updates and bazel cache</a> </h1> <div class="topic-category" itemscope itemtype="http://schema.org/BreadcrumbList"> <span itemprop="itemListElement" itemscope itemtype="http://schema.org/ListItem"> <a href="/c/tensorflow/special-interest-groups/build/49" class="badge-wrapper bullet" itemprop="item"> <span class='badge-category-bg' style='background-color: #f66f00'></span> <span class='badge-category clear-badge'> <span class='category-name' itemprop='name'>TensorFlow</span> </span> </a> <meta itemprop="position" content="1" /> </span> <span itemprop="itemListElement" itemscope itemtype="http://schema.org/ListItem"> <a href="/c/tensorflow/special-interest-groups/build/49" class="badge-wrapper bullet" itemprop="item"> <span class='badge-category-bg' style='background-color: #8e5de8'></span> <span class='badge-category clear-badge'> <span class='category-name' itemprop='name'>Special Interest Groups</span> </span> </a> <meta itemprop="position" content="2" /> </span> <span itemprop="itemListElement" itemscope itemtype="http://schema.org/ListItem"> <a href="/c/tensorflow/special-interest-groups/build/49" class="badge-wrapper bullet" itemprop="item"> <span class='badge-category-bg' style='background-color: #AF93E4'></span> <span class='badge-category clear-badge'> <span class='category-name' itemprop='name'>Build</span> </span> </a> <meta itemprop="position" content="3" /> </span> </div> <div class="topic-category"> <div class='discourse-tags list-tags'> <a href='https://discuss.ai.google.dev/tag/sig_mlir' class='discourse-tag' rel="tag">sig_mlir</a>, <a href='https://discuss.ai.google.dev/tag/sig_build' class='discourse-tag' rel="tag">sig_build</a>, <a href='https://discuss.ai.google.dev/tag/contributing' class='discourse-tag' rel="tag">contributing</a>, <a href='https://discuss.ai.google.dev/tag/community' class='discourse-tag' rel="tag">community</a>, <a href='https://discuss.ai.google.dev/tag/help_dev' class='discourse-tag' rel="tag">help_dev</a> </div> </div> </div> <div itemscope itemtype='http://schema.org/DiscussionForumPosting'> <meta itemprop='headline' content='LLVM updates and bazel cache'> <link itemprop='url' href='https://discuss.ai.google.dev/t/llvm-updates-and-bazel-cache/31793'> <meta itemprop='datePublished' content='2021-06-16T22:31:47Z'> <meta itemprop='articleSection' content='Build'> <meta itemprop='keywords' content='sig_mlir, sig_build, contributing, community, help_dev'> <div itemprop='publisher' itemscope itemtype="http://schema.org/Organization"> <meta itemprop='name' content='Build with Google AI'> <div itemprop='logo' itemscope itemtype="http://schema.org/ImageObject"> <meta itemprop='url' content='https://canada1.discourse-cdn.com/googleaitrial/original/1X/9c9b743566b5ef0d0701ebba72bf83038f42ea0b.png'> </div> </div> <div id='post_1' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Bhack'><span itemprop='name'>Bhack</span></a> </span> <link itemprop="mainEntityOfPage" href="https://discuss.ai.google.dev/t/llvm-updates-and-bazel-cache/31793"> <span class="crawler-post-infos"> <time datetime='2021-06-16T22:31:47Z' class='post-time'> June 16, 2021, 10:31pm </time> <meta itemprop='dateModified' content='2021-06-16T22:44:37Z'> <span itemprop='position'>1</span> </span> </div> <div class='post' itemprop='text'> <p>As we are updating LLVM <a href="https://github.com/tensorflow/tensorflow/commits/master/third_party/llvm">two times at day</a> I’ve tried to query <code>bazel</code> with:</p> <p><code>bazel aquery "rdeps(//tensorflow:*,//third_party/llvm:*)" --include_aspects 2>/dev/null | grep Compiling | wc -l</code></p> <p>I am not a <code>bazel</code> ninja so probably the query could be wrong or improved but I currently see <code>9938</code> files on master (CPU only).</p> <p>What is the effect of this bi-daily rolling update on the average community contributor compiling workflow/environment and his <code>bazel</code> cache?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_2' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Mehdi_AMINI'><span itemprop='name'>Mehdi_AMINI</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T00:42:59Z' class='post-time'> August 27, 2021, 12:42am </time> <meta itemprop='dateModified' content='2021-08-27T00:42:59Z'> <span itemprop='position'>2</span> </span> </div> <div class='post' itemprop='text'> <p>I’d assume that Bazel is smart enough to only recompile the pieces that actually changed in there, so the impact will vary depending on the actual updates we’re picking up.</p> <p>As a workflow, when I use to develop in LLVM on a laptop, I would have a cron script that would run a <code>git pull</code> and build at 7am before I show up in the office so that when I arrive I have the most recent copy of the code with the build cache up-to-date <img src="https://emoji.discourse-cdn.com/twitter/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_4' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Bhack'><span itemprop='name'>Bhack</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T09:28:29Z' class='post-time'> August 27, 2021, 9:28am </time> <meta itemprop='dateModified' content='2021-08-27T09:28:29Z'> <span itemprop='position'>4</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="Mehdi_AMINI" data-post="2" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/mehdi_amini/48/8112_2.png" class="avatar"> Mehdi_AMINI:</div> <blockquote> <p>I’d assume that Bazel is smart enough to only recompile the pieces that actually changed in there, so the impact will vary depending on the actual updates we’re picking up.</p> </blockquote> </aside> <p>If you execute the query on your laptop you will see that we have many targets depending on this. Are we going to invalidate all this targets cache?</p> <aside class="quote no-group" data-username="Mehdi_AMINI" data-post="2" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/mehdi_amini/48/8112_2.png" class="avatar"> Mehdi_AMINI:</div> <blockquote> <p>As a workflow, when I use to develop in LLVM on a laptop, I would have a cron script that would run a <code>git pull</code> and build at 7am before I show up in the office so that when I arrive I have the most recent copy of the code with the build cache up-to-date</p> </blockquote> </aside> <p>This could work for you or any other TF team member as a daily routine.</p> <p>But I think that it is really not the case of<br> an occasional/episodic contributor that invest his sparse time to contribute a TF PR.</p> <p>What do you think?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_6' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Mehdi_AMINI'><span itemprop='name'>Mehdi_AMINI</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T14:48:14Z' class='post-time'> August 27, 2021, 2:48pm </time> <meta itemprop='dateModified' content='2021-08-27T14:48:14Z'> <span itemprop='position'>6</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="Bhack" data-post="4" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/bhack/48/6388_2.png" class="avatar"> Bhack:</div> <blockquote> <p>If you execute the query on your laptop you will see that we have many targets depending on this. Are we going to invalidate all this targets cache?</p> </blockquote> </aside> <p>Bazel is supposed to cache thing based on the content hash of the files: so updating LLVM matters only for the actual files changed in there.</p> <aside class="quote no-group" data-username="Bhack" data-post="4" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/bhack/48/6388_2.png" class="avatar"> Bhack:</div> <blockquote> <p>This could work for you or any other TF team member as a daily routine.</p> <p>But I think that it is really not the case of<br> an occasional/episodic contributor that invest his sparse time to contribute a TF PR.</p> </blockquote> </aside> <p>Yes, but if you’re an occasional contributor, whether we update some code twice a day or every other week shouldn’t matter: you’ll have to (re)build it, won’t you?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_7' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Bhack'><span itemprop='name'>Bhack</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T14:56:08Z' class='post-time'> August 27, 2021, 2:56pm </time> <meta itemprop='dateModified' content='2021-08-27T14:56:08Z'> <span itemprop='position'>7</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="Mehdi_AMINI" data-post="6" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/mehdi_amini/48/8112_2.png" class="avatar"> Mehdi_AMINI:</div> <blockquote> <p>Yes, but if you’re an occasional contributor, whether we update some code twice a day or every other week shouldn’t matter: you’ll have to (re)build it, won’t you?</p> </blockquote> </aside> <p>Without an available <a href="https://github.com/tensorflow/build/issues/5">nightly cache</a> probably.<br> But it really depends on how many targets llvm will invalidate in the case the contribution is not too sparse.</p> <p>An llvm update Is It going to invalidate all the targets depending on llvm?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_8' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Mehdi_AMINI'><span itemprop='name'>Mehdi_AMINI</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T15:08:58Z' class='post-time'> August 27, 2021, 3:08pm </time> <meta itemprop='dateModified' content='2021-08-27T15:08:58Z'> <span itemprop='position'>8</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="Bhack" data-post="7" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/bhack/48/6388_2.png" class="avatar"> Bhack:</div> <blockquote> <p>An llvm update Is It going to invalidate all the targets depending on llvm?</p> </blockquote> </aside> <p>LLVM isn’t monolithic, it depends what is changed in LLVM. Bazel tracks finer grain dependencies. If LLVM LibSupport is changed, then most things will get rebuilt, but if there is a fix in the X86 backend I would expect only the JIT dependencies to be rebuilt for example.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_9' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Bhack'><span itemprop='name'>Bhack</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T15:24:11Z' class='post-time'> August 27, 2021, 3:24pm </time> <meta itemprop='dateModified' content='2021-08-27T15:24:11Z'> <span itemprop='position'>9</span> </span> </div> <div class='post' itemprop='text'> <p>I am not worried about LLVM itself but of all the targets in its chain if the query was correct.</p> <p>Is a small change in llvm going to invalidated all these targets?</p> <p><code>bazel aquery "rdeps(//tensorflow:*,//third_party/llvm:*)" --include_aspects 2>/dev/null | grep Compiling</code></p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_10' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/gcmn'><span itemprop='name'>gcmn</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T16:11:49Z' class='post-time'> August 27, 2021, 4:11pm </time> <meta itemprop='dateModified' content='2021-08-27T16:11:49Z'> <span itemprop='position'>10</span> </span> </div> <div class='post' itemprop='text'> <p>So as Mehdi was saying, you can’t just look at all TF dependencies on anything in LLVM: it’s going to matter what in LLVM changes. A small change in LLVM is going to invalidate things that depend on the <em>specific target</em> in LLVM that changed. That said, TF’s monolithic structure is pretty likely to create unnecessary dependencies and cause unnecessary recompiles.</p> <p>Your query also doesn’t work for me. Maybe because of recent changes to use the upstream LLVM Bazel build files? <code>//third_party/llvm</code> is empty. I think you want <code>@llvm-project</code>. Everything in the repo would be <code>@llvm-project//...</code>. Similarly, your query for <code>//tensorflow:*</code> is I believe only capturing rules directly in that package and you’d need <code>//tensorflow/...</code> to get everything. But for reasons that aren’t really clear to me doing an aquery with either of those wildcards fail in different ways. Fun. Anyway, so we’ll limit ourselves to individual packages for now.</p> <p>If llvm:Support changes:</p> <pre data-code-wrap="shell"><code class="lang-shell">$ bazel aquery "rdeps(//tensorflow:*, @llvm-project//llvm:Support)" --include_aspects 2>/dev/null | grep Compiling | wc -l 4930 </code></pre> <p>but if llvm:Symbolize changes, then nothing needs to be recompiled</p> <pre data-code-wrap="shell"><code class="lang-shell">$ bazel aquery "rdeps(//tensorflow:*, @llvm-project//llvm:Symbolize)" --include_aspects 2>/dev/null | grep Compiling | wc -l 0 </code></pre> <p>for an in-between example:</p> <pre data-code-wrap="shell"><code class="lang-shell">$ bazel aquery "rdeps(//tensorflow:*, @llvm-project//llvm:ProfileData)" --include_aspects 2>/dev/null | grep Compiling | wc -l 2818 </code></pre> <p>and don’t forget about MLIR:</p> <pre data-code-wrap="shell"><code class="lang-shell">$ bazel aquery "rdeps(//tensorflow:*, @llvm-project//mlir:Support)" --include_aspects 2>/dev/null | grep Compiling | wc -l 1924 </code></pre> <p>Another important component is that if you’re using a cache like <code>--disk_cache</code>, Bazel will only rerun a compile command if the inputs <em>actually</em> change because it’s hashing the content. So if you have a change to llvm:Support that adds something hidden behind a preprocessor macro but that doesn’t actually result in a different compiled output, then Bazel will not recompile things that depend on it, instead using the cache hit.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_11' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Bhack'><span itemprop='name'>Bhack</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T16:28:29Z' class='post-time'> August 27, 2021, 4:28pm </time> <meta itemprop='dateModified' content='2021-08-27T16:55:35Z'> <span itemprop='position'>11</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="gcmn" data-post="10" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/gcmn/48/6534_2.png" class="avatar"> gcmn:</div> <blockquote> <p>Maybe because of recent changes to use the upstream LLVM Bazel build files?</p> </blockquote> </aside> <p>Yes probably as the query was tested in June.</p> <aside class="quote no-group" data-username="gcmn" data-post="10" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/gcmn/48/6534_2.png" class="avatar"> gcmn:</div> <blockquote> <p>That said, TF’s monolithic structure is pretty likely to create unnecessary dependencies and cause unnecessary recompiles.</p> </blockquote> </aside> <p>This Is true but I am not shure how a so low-level components like LLVM and MLIR without version releases could be really isolated as modular dependecies.</p> <aside class="quote no-group" data-username="gcmn" data-post="10" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/gcmn/48/6534_2.png" class="avatar"> gcmn:</div> <blockquote> <p>Another important component is that if you’re using a cache like <code>--disk_cache</code></p> </blockquote> </aside> <p>To freeze TF only can you just analyze the <a href="https://docs.bazel.build/versions/4.2.0/remote-execution-caching-debug.html#ensure-re-running-the-same-buildtest-command-produces-cache-hits">cache hits missing</a> before ad after a few of llvm updated commits?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_12' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/gcmn'><span itemprop='name'>gcmn</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T16:57:31Z' class='post-time'> August 27, 2021, 4:57pm </time> <meta itemprop='dateModified' content='2021-08-27T16:57:31Z'> <span itemprop='position'>12</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group quote-modified" data-username="Bhack" data-post="11" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/bhack/48/6388_2.png" class="avatar"> Bhack:</div> <blockquote> <p>To freeze TF only can you just analyze the [cache hits missing](Probably It Is something like <a href="https://docs.bazel.build/versions/4.2.0/remote-execution-caching-debug.html#ensure-re-running-the-same-buildtest-command-produces-cache-hits">Debugging remote cache hits for remote execution - Bazel 4.2.0</a>) before ad after a few of llvm updated commits?</p> </blockquote> </aside> <p>I’m not really sure. But you could also look at the the command profile after an LLVM update: <a href="https://source.cloud.google.com/results/invocations/db44e648-2768-4e83-85fc-e63e092c880b/artifacts/">https://source.cloud.google.com/results/invocations/db44e648-2768-4e83-85fc-e63e092c880b/artifacts/</a></p> <p>Just <code>bazel analyze-profile</code> on that doesn’t offer any insight on cache hit rate but perhaps extracting the data with <code>--dump=raw</code> would.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_13' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/mihaimaruseac'><span itemprop='name'>mihaimaruseac</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T17:39:42Z' class='post-time'> August 27, 2021, 5:39pm </time> <meta itemprop='dateModified' content='2021-08-27T17:39:42Z'> <span itemprop='position'>13</span> </span> </div> <div class='post' itemprop='text'> <p>Just ran an experiment checking <a href="https://github.com/tensorflow/tensorflow/commit/f696b566f40baa87d776c92a12b03cca1d83bfd1">a commit</a> before the <a href="https://github.com/tensorflow/tensorflow/commit/a868b0d057b34dbd487a1e3d2b08d5489651b3ff">recent LLVM bump</a> on a newly created disk cache:</p> <pre data-code-wrap="bash"><code class="lang-bash">...$ git checkout f696b566f40baa87d776c92a12b03cca1d83bfd1 ...$ bazel clean --expunge ...$ time bazel build --disk_cache=~/bazel_cache //tensorflow/tools/pip_package:build_pip_package ... INFO: Elapsed time: 1176.812s, Critical Path: 251.94s INFO: 11114 processes: 1289 internal, 9825 local. INFO: Build completed successfully, 11114 total actions real 19m36.954s user 0m0.359s sys 0m0.380s </code></pre> <p>Next, switch to the LLVM bump and compile again:</p> <pre data-code-wrap="bash"><code class="lang-bash">...$ git checkout a868b0d057b34dbd487a1e3d2b08d5489651b3ff ...$ time bazel build --disk_cache=~/bazel_cache //tensorflow/tools/pip_package:build_pip_package ... INFO: Elapsed time: 523.303s, Critical Path: 208.88s INFO: 3273 processes: 166 internal, 3107 local. INFO: Build completed successfully, 3273 total actions real 8m43.377s user 0m0.166s sys 0m0.171s </code></pre> <p>Note, this is on a fast machine, OSS contributors likely won’t have access to these specs:</p> <pre data-code-wrap="bash"><code class="lang-bash">...$ cat /proc/cpuinfo # 56 CPUs ... processor : 55 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz stepping : 1 microcode : 0xb00003e cpu MHz : 3192.602 cache size : 35840 KB ... ...$ free -h # 126 GB RAM total used free shared buff/cache available Mem: 125Gi 42Gi 61Gi 1.5Gi 21Gi 80Gi Swap: 127Gi 741Mi 127Gi </code></pre> <p>This experiment could be repeated for other LLVM bumps (or TFRT ones too).</p> <p>So, the most recent LLVM bump caused a recompile of almost 50% (time-wise) of a full compile. This impacts anyone developing in OSS but also prevents us from using caching and GitHub Actions for GitHub presubmits. Maybe both of our teams can invest some time to reduce this (assuming this happens on other LLVM bumps)?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_14' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Bhack'><span itemprop='name'>Bhack</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T17:47:52Z' class='post-time'> August 27, 2021, 5:47pm </time> <meta itemprop='dateModified' content='2021-08-27T17:47:52Z'> <span itemprop='position'>14</span> </span> </div> <div class='post' itemprop='text'> <p>Thank you <a class="mention" href="/u/mihaimaruseac">@mihaimaruseac</a> for this small test on a random lllvm udpate.</p> <p>Also occasional contributors will not reach this resources configuration also with the <a href="https://github.com/features/codespaces">top Codespaces profile</a> (32 cores).</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_15' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/mihaimaruseac'><span itemprop='name'>mihaimaruseac</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T18:31:46Z' class='post-time'> August 27, 2021, 6:31pm </time> <meta itemprop='dateModified' content='2021-08-27T18:31:46Z'> <span itemprop='position'>15</span> </span> </div> <div class='post' itemprop='text'> <p>A different set of commits, a new test:</p> <pre data-code-wrap="bash"><code class="lang-bash">...$ git checkout b836deac35cd58c271aebbebdc6b0bd13a058585 ...$ rm -rf ~/bazel_cache/ ...$ bazel clean --expunge ...$ time bazel build --disk_cache=~/bazel_cache //tensorflow/tools/pip_package:build_pip_package ... INFO: Elapsed time: 1164.101s, Critical Path: 308.54s INFO: 11114 processes: 1289 internal, 9825 local. INFO: Build completed successfully, 11114 total actions real 19m24.243s user 0m0.360s sys 0m0.336s ...$ git checkout b71106370c45bd584ffbdde02be21d35b882d9ee Previous HEAD position was b836deac35c Remove TensorShape dependency from ScopedMemoryDebugAnnotation. HEAD is now at b71106370c4 Integrate LLVM at llvm/llvm-project@bd7ece4e063e ...$ time bazel build --disk_cache=~/bazel_cache //tensorflow/tools/pip_package:build_pip_package ... INFO: Elapsed time: 262.158s, Critical Path: 120.47s INFO: 1567 processes: 149 internal, 1418 local. INFO: Build completed successfully, 1567 total actions real 4m22.240s user 0m0.095s sys 0m0.092s </code></pre> <p>This is around <a href="https://github.com/tensorflow/tensorflow/commit/b71106370c45bd584ffbdde02be21d35b882d9ee">LLVM bump from 23 hours ago</a>, so the two tests run so far cover the bumps that occur in a single day.</p> <p>This is slightly faster that before, only at 20% of a full compile.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_16' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Bhack'><span itemprop='name'>Bhack</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-27T23:13:51Z' class='post-time'> August 27, 2021, 11:13pm </time> <meta itemprop='dateModified' content='2021-08-27T23:13:51Z'> <span itemprop='position'>16</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-tfteam" data-username="mihaimaruseac" data-post="13" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/mihaimaruseac/48/6253_2.png" class="avatar"> mihaimaruseac:</div> <blockquote> <p>This experiment could be repeated for other LLVM bumps (or TFRT ones too).</p> </blockquote> </aside> <p>It think that all these bumps, with this cache miss rate, are going to impact also the <code>disk-cache</code> size o. disk as It could be quite common to have more then one branch/PR waiting for a review and so you need to switch forth and back between branches on different bumps (or probably need we to costantly rebase/merge when possibile?).</p> <p>As we know clean up the <code>disk-cache</code> over time is really not a so straightforward task:</p> <aside class="onebox githubissue" data-onebox-src="https://github.com/bazelbuild/bazel/issues/5139"> <header class="source"> <a href="https://github.com/bazelbuild/bazel/issues/5139" target="_blank" rel="noopener">github.com/bazelbuild/bazel</a> </header> <article class="onebox-body"> <div class="github-row"> <div class="github-icon-container" title="Issue"> <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg> </div> <div class="github-info-container"> <h4> <a href="https://github.com/bazelbuild/bazel/issues/5139" target="_blank" rel="noopener">Implement automatic garbage collection for the disk cache</a> </h4> <div class="github-info"> <div class="date"> opened <span class="discourse-local-date" data-format="ll" data-date="2018-05-02" data-time="11:28:03" data-timezone="UTC">11:28AM - 02 May 18 UTC</span> </div> <div class="user"> <a href="https://github.com/buchgr" target="_blank" rel="noopener"> <img alt="buchgr" src="https://canada1.discourse-cdn.com/googleaitrial/original/2X/9/9f453a7fd0d39b43b60d3e704c57d821ebfbee66.jpeg" class="onebox-avatar-inline" width="20" height="20" data-dominant-color="866351"> buchgr </a> </div> </div> <div class="labels"> <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;"> type: feature request </span> <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;"> P2 </span> <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;"> team-Remote-Exec </span> </div> </div> </div> <div class="github-row"> <p class="github-body-container">Break out from https://github.com/bazelbuild/bazel/issues/4870. Bazel can use<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden"> a local directory as a remote cache via the `--disk_cache` flag. We want it to also be able to automatically clean the cache after a size threshold has been reached. It probably makes sense to clean based on least recently used semantics. @RNabel would you want to work on this? @RNabel @davido</span></p> </div> </article> <div class="onebox-metadata"> </div> <div style="clear: both"></div> </aside> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_17' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/mihaimaruseac'><span itemprop='name'>mihaimaruseac</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-28T21:13:52Z' class='post-time'> August 28, 2021, 9:13pm </time> <meta itemprop='dateModified' content='2021-08-28T21:25:34Z'> <span itemprop='position'>17</span> </span> </div> <div class='post' itemprop='text'> <p>Actually, let’s run a longer experiment and track changes since <a href="https://github.com/tensorflow/tensorflow/commit/699c63cf6b0136a330ae8c5f56a2087361f6701e">the first commit of this week</a>:</p> <pre data-code-wrap="bash"><code class="lang-bash">...$ bazel clean --expunge ...$ rm -rf ~/bazel_cache/ ...$ for commit in $(git log --first-parent --pretty=oneline ...623ed300b593b368e665899be3cf080c5a0e3ebe | tac | cut -d' ' -f1) > do > git checkout ${commit} &> /dev/null > echo "Building at ${commit}" > (time bazel build --disk_cache=~/bazel_cache //tensorflow/tools/pip_package:build_pip_package >/dev/null) 2>&1 | grep real > done | tee ~/build_log </code></pre> <p>This is 273 commits. At the end of the job, the build cache has 124GB (something most people in OSS cannot afford either):</p> <pre data-code-wrap="bash"><code class="lang-bash">...$ du -sh ~/bazel_cache/ 124G /usr/local/google/home/mihaimaruseac/bazel_cache/ </code></pre> <p>Anyway, let’s look at the timing info from the log:</p> <pre data-code-wrap="bash"><code class="lang-bash"># concatenate the lines, transform xmys time into (60*x+y) seconds ...$ awk '!(NR%2) { print p, $2} { p = $3 }' ~/build_log | sed -e 's/m/ /' -e 's/s$//' | awk '{ print $1, ($2 * 60 + $3) }' > ~/times_in_seconds # get a histogram binned for every 10 seconds ...$ sort -rnk2 ~/times_in_seconds | cut -d. -f1 | cut -d' ' -f2 | sed -e 's/.$//' | uniq -c | perl -lane 'print $F[1], "0-", $F[1], "9\t", "=" x ($F[0] / 2)' 1180-1189 580-589 570-579 550-559 = 520-529 = 430-439 360-369 280-289 270-279 240-249 230-239 210-219 = 170-179 = 160-169 = 140-149 120-129 == 110-119 = 90-99 80-89 = 70-79 === 60-69 == 50-59 === 40-49 ========== 30-39 ===================================================================================================== # also print the values instead of just ====s ...$ sort -rnk2 ~/times_in_seconds | cut -d. -f1 | cut -d' ' -f2 | sed -e 's/.$//' | uniq -c 1 118 1 58 1 57 2 55 2 52 1 43 1 36 1 28 1 27 1 24 1 23 2 21 2 17 3 16 1 14 4 12 3 11 1 9 3 8 7 7 5 6 7 5 20 4 202 3 </code></pre> <p>As you see, most incremental builds take 30-40 seconds (202 out of 273!) but there are some that take much longer. Let’s look into them</p> <pre data-code-wrap="bash"><code class="lang-bash"># longest 20 times ...$ sort -rnk2 ~/times_in_seconds | head -n20 699c63cf6b0136a330ae8c5f56a2087361f6701e 1184.46 b836deac35cd58c271aebbebdc6b0bd13a058585 585.35 abcced051cb1bd8fb05046ac3b6023a7ebcc4578 574.188 b49d731332e5d9929acc9bfc9aed88ace61b6d18 556.711 42f72014a24e218a836a87452885359919866b0b 553.296 982608d75d1493b4e351ef84d58bc0fdf78203c8 527.231 a868b0d057b34dbd487a1e3d2b08d5489651b3ff 523.162 c432f62159879d83e62d72afc9ef80cb6cdbe1e5 433.18 8b05b58c7c9cb8d1ed838a3157ddda8694c028f4 366.548 36931bae2a36efda71f96c9e879e91b087874e89 280.591 b71106370c45bd584ffbdde02be21d35b882d9ee 272.807 86fb36271f9068f84ddcecae74fe0b7df9ce83ee 242.273 1848375d184177741de4dfa4b65e497b868283cd 239.788 9770c84ea45587524e16de233d3cf8b258a9bd77 219.21 61bcb9df099b3be7dfbbbba051ca007032bfb777 214.006 d3a17786019d534fb7a112dcda5583b8fd6e7a62 172.092 e8dc63704c88007ee4713076605c90188d66f3d2 170.582 ddcc48f003e6fe233a6d63d3d3f5fde9f17404f1 169.959 2035c4acc478b475c149f9be4f2209531d3d2d0d 169.84 3edbbc918a940162fc9ae4d69bba0fff86db9ca2 167.948 # what are the commits for each one ...$ for commit in $(sort -rnk2 ~/times_in_seconds | head -n20 | awk '{ print $1 }'); do git log -n1 --pretty=oneline ${commit}; done 699c63cf6b0136a330ae8c5f56a2087361f6701e use tensorflow==2.5.0 to temporarily solve the failure of `evaluate_tflite` function. b836deac35cd58c271aebbebdc6b0bd13a058585 Remove TensorShape dependency from ScopedMemoryDebugAnnotation. abcced051cb1bd8fb05046ac3b6023a7ebcc4578 Prevent crashes when loading tensor slices with unsupported types. b49d731332e5d9929acc9bfc9aed88ace61b6d18 Integrate LLVM at llvm/llvm-project@955b91c19c00 42f72014a24e218a836a87452885359919866b0b Remove experimental flag `fetch_remote_devices_in_multi_client`. 982608d75d1493b4e351ef84d58bc0fdf78203c8 Switched to OSS llvm build rules instead of scripts imported from third_party. a868b0d057b34dbd487a1e3d2b08d5489651b3ff Integrate LLVM at llvm/llvm-project@fe611b1da84b c432f62159879d83e62d72afc9ef80cb6cdbe1e5 Integrate LLVM at llvm/llvm-project@b52171629f56 8b05b58c7c9cb8d1ed838a3157ddda8694c028f4 Integrate LLVM at llvm/llvm-project@8c3886b0ec98 36931bae2a36efda71f96c9e879e91b087874e89 Integrate LLVM at llvm/llvm-project@4b4bc1ea16de b71106370c45bd584ffbdde02be21d35b882d9ee Integrate LLVM at llvm/llvm-project@bd7ece4e063e 86fb36271f9068f84ddcecae74fe0b7df9ce83ee Integrate LLVM at llvm/llvm-project@fda176892e64 1848375d184177741de4dfa4b65e497b868283cd Merge pull request #51511 from PragmaTwice:patch-1 9770c84ea45587524e16de233d3cf8b258a9bd77 Integrate LLVM at llvm/llvm-project@cc4bfd7f59d5 61bcb9df099b3be7dfbbbba051ca007032bfb777 Integrate LLVM at llvm/llvm-project@8e284be04f2c d3a17786019d534fb7a112dcda5583b8fd6e7a62 Fix and resubmit subgroup change e8dc63704c88007ee4713076605c90188d66f3d2 Add BuildTensorSlice for building from unvalidated TensorSliceProtos. ddcc48f003e6fe233a6d63d3d3f5fde9f17404f1 [XLA:SPMD] Improve partial manual sharding handling. - Main change: make sharding propagation work natively with manual subgroup sharding. There were some problems when propagating with tuple shapes. This also avoids many copies, which is important for performance since the pass runs multiple times. - Normalize HloSharding::Subgroup() to merge the same type of subgroup dims. - Handle tuple-shaped ops (e.g., argmax as reduce, sort) in SPMD partitioner. - Make SPMD partitioner to handle pass-through ops (e.g., tuple) natively, since they can mix partial and non-partial elements in a tuple. 2035c4acc478b475c149f9be4f2209531d3d2d0d Legalizes GatherOp via canonicalization to GatherV2Op; i.e. Providing default values of 0 for the axis parameter and the batch_dims attribute. 3edbbc918a940162fc9ae4d69bba0fff86db9ca2 Internal change </code></pre> <p>10 of these 20 commits are LLVM hash bumps. In total, there are 11 such commits in the 273 considered:</p> <pre data-code-wrap="bash"><code class="lang-bash">...$ for commit in $(cat ~/times_in_seconds | awk '{ print $1 }'); do git log -n1 --pretty=oneline ${commit}; done | grep LLVM | wc -l 11 </code></pre> <p>So, <strong>almost all LLVM commits result in large compile times</strong>. Half of the top 20 longest compile times are LLVM hash bumps</p> <p>I’d say this is quite costly and we need to find a plan to handle this in a way that helps OSS users.</p> <p>Edit: Actually <strong>ALL</strong> LLVM hash bumps are included in the longest compiles, the missing one is just the conversion to upstream files:</p> <pre data-code-wrap="bash"><code class="lang-bash">...$ for commit in $(cat ~/times_in_seconds | awk '{ print $1 }'); do git log -n1 --pretty=oneline ${commit}; done | grep LLVM b49d731332e5d9929acc9bfc9aed88ace61b6d18 Integrate LLVM at llvm/llvm-project@955b91c19c00 3487b91d529f2cbc412121d60845cda014e0db7d Integrate LLVM at llvm/llvm-project@9cdd4ea06f09 c432f62159879d83e62d72afc9ef80cb6cdbe1e5 Integrate LLVM at llvm/llvm-project@b52171629f56 86fb36271f9068f84ddcecae74fe0b7df9ce83ee Integrate LLVM at llvm/llvm-project@fda176892e64 36931bae2a36efda71f96c9e879e91b087874e89 Integrate LLVM at llvm/llvm-project@4b4bc1ea16de e624ad903f9c796a98bd309268ccfca5e7a9c19a Use upstream LLVM Bazel build rules 8b05b58c7c9cb8d1ed838a3157ddda8694c028f4 Integrate LLVM at llvm/llvm-project@8c3886b0ec98 9770c84ea45587524e16de233d3cf8b258a9bd77 Integrate LLVM at llvm/llvm-project@cc4bfd7f59d5 b71106370c45bd584ffbdde02be21d35b882d9ee Integrate LLVM at llvm/llvm-project@bd7ece4e063e a868b0d057b34dbd487a1e3d2b08d5489651b3ff Integrate LLVM at llvm/llvm-project@fe611b1da84b 61bcb9df099b3be7dfbbbba051ca007032bfb777 Integrate LLVM at llvm/llvm-project@8e284be04f2c </code></pre> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_18' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Bhack'><span itemprop='name'>Bhack</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-28T23:43:27Z' class='post-time'> August 28, 2021, 11:43pm </time> <meta itemprop='dateModified' content='2021-08-28T23:51:42Z'> <span itemprop='position'>18</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-tfteam" data-username="mihaimaruseac" data-post="17" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/mihaimaruseac/48/6253_2.png" class="avatar"> mihaimaruseac:</div> <blockquote> <p>So, <strong>almost all LLVM commits result in large compile times</strong>. Half of the top 20 longest compile times are LLVM hash bumps</p> <p>I’d say this is quite costly and we need to find a plan to handle this in a way that helps OSS users.</p> </blockquote> </aside> <p>Thank you for this extended analysis.<br> Can you limit the number of cores/ram on one of these builds just to understand what we are talking about on an expected OSS user hw configuration?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_19' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/mihaimaruseac'><span itemprop='name'>mihaimaruseac</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-29T01:52:17Z' class='post-time'> August 29, 2021, 1:52am </time> <meta itemprop='dateModified' content='2021-08-29T01:52:17Z'> <span itemprop='position'>19</span> </span> </div> <div class='post' itemprop='text'> <p>I can run the experiment at home from a different computer. But only compiling before and after a LLVM bump, I’ll pick one that was already considered in the thread</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_20' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Bhack'><span itemprop='name'>Bhack</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-29T10:06:37Z' class='post-time'> August 29, 2021, 10:06am </time> <meta itemprop='dateModified' content='2021-08-29T19:39:44Z'> <span itemprop='position'>20</span> </span> </div> <div class='post' itemprop='text'> <p>I don’t know if you can<br> constrain It with these args is it enought also on you standard build machine:</p> <p><a href="https://docs.bazel.build/versions/main/user-manual.html#flag--local_%7Bram,cpu%7D_resources" class="onebox" target="_blank" rel="noopener">https://docs.bazel.build/versions/main/user-manual.html#flag--local_{ram,cpu}_resources</a></p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_22' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Jacques_Pienaar'><span itemprop='name'>Jacques_Pienaar</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-30T15:04:23Z' class='post-time'> August 30, 2021, 3:04pm </time> <meta itemprop='dateModified' content='2021-08-30T15:04:23Z'> <span itemprop='position'>22</span> </span> </div> <div class='post' itemprop='text'> <p>I think this is a nice start, but we need to go deeper: how much of the recompilation is due to which target dependencies changing? E.g., if we didn’t have the monolith :core target, what would have needed recompilation vs what is recompiled today? Cache only works if the dependencies are detangled, if independent parts are intermingled for historical reasons then there’ll be a lot of rebuilding. Also, how does the proposed work on making shared libraries support proper affect this? (These may be intermingled as tot get the best shared library, one would need better deps, but if statistically all linked together then cache hits will be lower and link time higher).</p> <p>But I think that goes to the above question: who is the user that this affecting? E.g., using the git development model, I’d be on a branch getting my work done up until sending for review and would hit this only if I manually pull (so I choose the frequency). At the point where I do that (and I think Mihai’s numbers may be with a RBE as I can’t built TF under an hour on my machine without a populated cache), I context switch and work on something else done. So updating affects a user that is pulling frequently for some reason, but not quickly enough to get enough cache reuse?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="2" /> <span class='post-likes'>2 Likes</span> </div> </div> <div id='post_23' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://discuss.ai.google.dev/u/Bhack'><span itemprop='name'>Bhack</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-08-30T15:20:18Z' class='post-time'> August 30, 2021, 3:20pm </time> <meta itemprop='dateModified' content='2021-08-30T15:20:18Z'> <span itemprop='position'>23</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-tfteam" data-username="Jacques_Pienaar" data-post="22" data-topic="31793"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://yyz1.discourse-cdn.com/googleaitrial/user_avatar/discuss.ai.google.dev/jacques_pienaar/48/6539_2.png" class="avatar"> Jacques_Pienaar:</div> <blockquote> <p>So updating affects a user that is pulling frequently for some reason, but not quickly enough to get enough cache reuse</p> </blockquote> </aside> <p>I think It will affect external contributors that have more then one waiting PR for a review or when they need to rebase as we have conflict against master.<br> These two cases really depend on how fast our review process is vs the number of bumps and the internal “direct commit/copybara” activities.</p> <p>Also it will affect when I contribute a new PR e.g. just a week or few days later to the last merged one.</p> <p>Then, but this a separated topic, it is also about what we are asking to the external contributor to execute a clean room build to populate the first cache and how to manage the garbage collection for the <code>--disk_cache</code> as It is currently unmaneged in bazel specially if we are growing too fast on the disk size with these bumps.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> </div> <div role='navigation' itemscope itemtype='http://schema.org/SiteNavigationElement' class="topic-body crawler-post"> <span itemprop='name'><b><a rel="next" itemprop="url" href="/t/llvm-updates-and-bazel-cache/31793?page=2">next page →</a></b></span> </div> <div id="related-topics" class="more-topics__list " role="complementary" aria-labelledby="related-topics-title"> <h3 id="related-topics-title" class="more-topics__list-title"> Related topics </h3> <div class="topic-list-container" itemscope itemtype='http://schema.org/ItemList'> <meta itemprop='itemListOrder' content='http://schema.org/ItemListOrderDescending'> <table class='topic-list'> <thead> <tr> <th>Topic</th> <th></th> <th class="replies">Replies</th> <th class="views">Views</th> <th>Activity</th> </tr> </thead> <tbody> <tr class="topic-list-item" id="topic-list-item-27937"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='1'> <span class="link-top-line"> <a itemprop='url' href='https://discuss.ai.google.dev/t/meeting-notes-from-9-22-2022/27937' class='title raw-link raw-topic-link'>Meeting notes from 9/22/2022</a> </span> <div class="link-bottom-line"> <a href='/c/tensorflow/special-interest-groups/sig-federated/53' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #AF93E4'></span> <span class='badge-category clear-badge'> <span class='category-name'>SIG Federated</span> </span> </a> <div class="discourse-tags"> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>0</span> </td> <td class="views"> <span class='views' title='views'>734</span> </td> <td> September 23, 2022 </td> </tr> <tr class="topic-list-item" id="topic-list-item-29558"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='2'> <span class="link-top-line"> <a itemprop='url' href='https://discuss.ai.google.dev/t/so-footprint-reduction-not-working-2-6-0/29558' class='title raw-link raw-topic-link'>.so footprint reduction not working (2.6.0)</a> </span> <div class="link-bottom-line"> <a href='/c/tensorflow/general-discussion-6/37' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #f2a346'></span> <span class='badge-category clear-badge'> <span class='category-name'>General Discussion</span> </span> </a> <div class="discourse-tags"> <a href='https://discuss.ai.google.dev/tag/tflite' class='discourse-tag'>tflite</a> , <a href='https://discuss.ai.google.dev/tag/help_request' class='discourse-tag'>help_request</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>5</span> </td> <td class="views"> <span class='views' title='views'>937</span> </td> <td> September 29, 2021 </td> </tr> <tr class="topic-list-item" id="topic-list-item-25862"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='3'> <span class="link-top-line"> <a itemprop='url' href='https://discuss.ai.google.dev/t/sig-build-september-meeting-september-7-today-2pm/25862' class='title raw-link raw-topic-link'>SIG Build September Meeting: September 7 (today!) @ 2pm</a> </span> <div class="link-bottom-line"> <a href='/c/tensorflow/special-interest-groups/build/49' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #AF93E4'></span> <span class='badge-category clear-badge'> <span class='category-name'>Build</span> </span> </a> <div class="discourse-tags"> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>1</span> </td> <td class="views"> <span class='views' title='views'>1003</span> </td> <td> September 8, 2021 </td> </tr> <tr class="topic-list-item" id="topic-list-item-28439"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='4'> <span class="link-top-line"> <a itemprop='url' href='https://discuss.ai.google.dev/t/kudos-to-pluggabledevices-team-question-about-amd-gpu/28439' class='title raw-link raw-topic-link'>Kudos to PluggableDevices team; question about AMD GPU</a> </span> <div class="link-bottom-line"> <a href='/c/tensorflow/general-discussion-6/37' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #f2a346'></span> <span class='badge-category clear-badge'> <span class='category-name'>General Discussion</span> </span> </a> <div class="discourse-tags"> <a href='https://discuss.ai.google.dev/tag/tfdata' class='discourse-tag'>tfdata</a> , <a href='https://discuss.ai.google.dev/tag/pluggable_device' class='discourse-tag'>pluggable_device</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>20</span> </td> <td class="views"> <span class='views' title='views'>3254</span> </td> <td> December 14, 2021 </td> </tr> <tr class="topic-list-item" id="topic-list-item-38416"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='5'> <span class="link-top-line"> <a itemprop='url' href='https://discuss.ai.google.dev/t/newer-protobuf-grpc-bazel-and-bzlmod-support-plans/38416' class='title raw-link raw-topic-link'>Newer protobuf / gRPC / Bazel and Bzlmod support plans</a> </span> <div class="link-bottom-line"> <a href='/c/tensorflow/21' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #f66f00'></span> <span class='badge-category clear-badge'> <span class='category-name'>TensorFlow</span> </span> </a> <div class="discourse-tags"> <a href='https://discuss.ai.google.dev/tag/mediapipe' class='discourse-tag'>mediapipe</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>1</span> </td> <td class="views"> <span class='views' title='views'>65</span> </td> <td> September 16, 2024 </td> </tr> </tbody> </table> </div> </div> </div> <footer class="container wrap"> <nav class='crawler-nav'> <ul> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/' itemprop="url">Home </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/categories' itemprop="url">Categories </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/guidelines' itemprop="url">Guidelines </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='https://policies.google.com/terms' itemprop="url">Terms of Service </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='https://policies.google.com/privacy' itemprop="url">Privacy Policy </a> </span> </li> </ul> </nav> <p class='powered-by-link'>Powered by <a href="https://www.discourse.org">Discourse</a>, best viewed with JavaScript enabled</p> </footer> <div class="buorg"><div>Unfortunately, <a href="https://www.discourse.org/faq/#browser">your browser is unsupported</a>. Please <a href="https://browsehappy.com">switch to a supported browser</a> to view rich content, log in and reply.</div></div> </body> </html>