CINXE.COM
What extras names are treated as equal and why? - Packaging - Discussions on Python.org
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>What extras names are treated as equal and why? - Packaging - Discussions on Python.org</title> <meta name="description" content="Hello. In Fedora, we’ve recently started to deal with Python extras. For Python package names, we already generate RPM Provides and Requires in a form of: python3.Xdist(&lt;normalized_package_name&gt;) E.g.: python3.9dist(s&hellip;"> <meta name="generator" content="Discourse 3.5.0.beta3-dev - https://github.com/discourse/discourse version aaa648ceacc48dadf338d5f8e5d11ddb7ed63569"> <link rel="icon" type="image/png" href="https://us1.discourse-cdn.com/flex002/uploads/python1/optimized/1X/9997f0605d56c4bfecd63594f52f42cdafd6b06a_2_32x32.png"> <link rel="apple-touch-icon" type="image/png" href="https://us1.discourse-cdn.com/flex002/uploads/python1/optimized/1X/4c06143de7870c35963b818b15b395092a434991_2_180x180.png"> <meta name="theme-color" media="(prefers-color-scheme: light)" content="#fff"> <meta name="theme-color" media="(prefers-color-scheme: dark)" content="#111111"> <meta name="color-scheme" content="light dark"> <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0, viewport-fit=cover"> <link rel="canonical" href="https://discuss.python.org/t/what-extras-names-are-treated-as-equal-and-why/7614" /> <link rel="search" type="application/opensearchdescription+xml" href="https://discuss.python.org/opensearch.xml" title="Discussions on Python.org Search"> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/color_definitions_base__2_c2a804e3fad040873319ee4cd9c3c45781de3eca.css?__ws=discuss.python.org" media="(prefers-color-scheme: light)" rel="stylesheet" class="light-scheme"/><link href="https://sea2.discourse-cdn.com/flex002/stylesheets/color_definitions_dark_1_2_a79953e3d0e709f94450954b87894480f6ce6930.css?__ws=discuss.python.org" media="(prefers-color-scheme: dark)" rel="stylesheet" class="dark-scheme" data-scheme-id="1"/> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/desktop_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="desktop" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/automation_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="automation" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/checklist_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="checklist" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-ai_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-ai" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-akismet_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-akismet" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-chat-integration_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-chat-integration" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-data-explorer_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-data-explorer" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-details_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-details" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-lazy-videos_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-lazy-videos" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-local-dates_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-local-dates" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-math_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-math" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-narrative-bot_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-narrative-bot" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-policy_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-policy" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-presence_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-presence" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-solved_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-solved" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-templates_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-templates" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-topic-voting_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-topic-voting" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-user-notes_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-user-notes" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/footnote_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="footnote" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/hosted-site_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="hosted-site" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/poll_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="poll" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/spoiler-alert_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="spoiler-alert" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-ai_desktop_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-ai_desktop" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-topic-voting_desktop_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-topic-voting_desktop" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/poll_desktop_c02ae4694b7feffe7f15bebdb3958d5842ad5a7c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="poll_desktop" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/desktop_theme_4_78606131228b0c70c2a65f966e348a898fd2692c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="4" data-theme-name="unformatted code detector"/> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/desktop_theme_2_416172ebc02e278f8716d8de1b0c4ed72ca6d4c3.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="2" data-theme-name="light"/> <link rel="alternate nofollow" type="application/rss+xml" title="RSS feed of 'What extras names are treated as equal and why?'" href="https://discuss.python.org/t/what-extras-names-are-treated-as-equal-and-why/7614.rss" /> <meta property="og:site_name" content="Discussions on Python.org" /> <meta property="og:type" content="website" /> <meta name="twitter:card" content="summary" /> <meta name="twitter:image" content="https://us1.discourse-cdn.com/flex002/uploads/python1/original/1X/f93ff97c4f381b5e8add5a0c163b4ded29f20ed7.png" /> <meta property="og:image" content="https://us1.discourse-cdn.com/flex002/uploads/python1/original/1X/f93ff97c4f381b5e8add5a0c163b4ded29f20ed7.png" /> <meta property="og:url" content="https://discuss.python.org/t/what-extras-names-are-treated-as-equal-and-why/7614" /> <meta name="twitter:url" content="https://discuss.python.org/t/what-extras-names-are-treated-as-equal-and-why/7614" /> <meta property="og:title" content="What extras names are treated as equal and why?" /> <meta name="twitter:title" content="What extras names are treated as equal and why?" /> <meta property="og:description" content="Hello. In Fedora, we’ve recently started to deal with Python extras. For Python package names, we already generate RPM Provides and Requires in a form of: python3.Xdist(<normalized_package_name>) E.g.: python3.9dist(sectuptools-scm) Those are used on RPM level to bring in the required packages. It works reasonably well. We follow the normalizing rules from PEP 503. It is crucial to us that the names are normalized, because on RPM level, sectuptools-scm != sectuptools_scm. By using the norma..." /> <meta name="twitter:description" content="Hello. In Fedora, we’ve recently started to deal with Python extras. For Python package names, we already generate RPM Provides and Requires in a form of: python3.Xdist(<normalized_package_name>) E.g.: python3.9dist(sectuptools-scm) Those are used on RPM level to bring in the required packages. It works reasonably well. We follow the normalizing rules from PEP 503. It is crucial to us that the names are normalized, because on RPM level, sectuptools-scm != sectuptools_scm. By using the norma..." /> <meta property="og:article:section" content="Packaging" /> <meta property="og:article:section:color" content="ED207B" /> <meta name="twitter:label1" value="Reading time" /> <meta name="twitter:data1" value="9 mins 🕑" /> <meta name="twitter:label2" value="Likes" /> <meta name="twitter:data2" value="28 ❤" /> <meta property="article:published_time" content="2021-03-10T18:42:56+00:00" /> <meta property="og:ignore_canonical" content="true" /> <link rel="next" href="/t/what-extras-names-are-treated-as-equal-and-why/7614?page=2"> </head> <body class="crawler browser-update"> <header> <a href="/"> Discussions on Python.org </a> </header> <div id="main-outlet" class="wrap" role="main"> <div id="topic-title"> <h1> <a href="/t/what-extras-names-are-treated-as-equal-and-why/7614">What extras names are treated as equal and why?</a> </h1> <div class="topic-category" itemscope itemtype="http://schema.org/BreadcrumbList"> <span itemprop="itemListElement" itemscope itemtype="http://schema.org/ListItem"> <a href="/c/packaging/14" class="badge-wrapper bullet" itemprop="item"> <span class='badge-category-bg' style='background-color: #ED207B'></span> <span class='badge-category clear-badge'> <span class='category-name' itemprop='name'>Packaging</span> </span> </a> <meta itemprop="position" content="1" /> </span> </div> </div> <div itemscope itemtype='http://schema.org/DiscussionForumPosting'> <meta itemprop='headline' content='What extras names are treated as equal and why?'> <link itemprop='url' href='https://discuss.python.org/t/what-extras-names-are-treated-as-equal-and-why/7614'> <meta itemprop='datePublished' content='2021-03-10T18:42:56Z'> <meta itemprop='articleSection' content='Packaging'> <meta itemprop='keywords' content=''> <div itemprop='publisher' itemscope itemtype="http://schema.org/Organization"> <meta itemprop='name' content='Python Software Foundation'> <div itemprop='logo' itemscope itemtype="http://schema.org/ImageObject"> <meta itemprop='url' content='https://us1.discourse-cdn.com/flex002/uploads/python1/original/1X/c7591c98caf3b31d4d9c6f322f41ed9d80a50800.png'> </div> </div> <div id='post_1' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/hroncok'><span itemprop='name'>hroncok</span></a> (Miro Hrončok) </span> <link itemprop="mainEntityOfPage" href="https://discuss.python.org/t/what-extras-names-are-treated-as-equal-and-why/7614"> <span class="crawler-post-infos"> <time datetime='2021-03-10T18:42:56Z' class='post-time'> March 10, 2021, 6:42pm </time> <meta itemprop='dateModified' content='2021-03-10T18:42:56Z'> <span itemprop='position'>1</span> </span> </div> <div class='post' itemprop='text'> <p>Hello. In Fedora, we’ve recently <a href="https://fedoraproject.org/wiki/Changes/PythonExtras" rel="noopener nofollow ugc">started to deal</a> with <a href="https://www.python.org/dev/peps/pep-0508/#extras" rel="noopener nofollow ugc">Python extras</a>. For Python package names, we already generate RPM <em>Provides</em> and <em>Requires</em> in a form of:</p> <pre><code>python3.Xdist(<normalized_package_name>) </code></pre> <p>E.g.:</p> <pre><code>python3.9dist(sectuptools-scm) </code></pre> <p>Those are used on RPM level to bring in the required packages. It works reasonably well.</p> <p>We follow the <a href="https://www.python.org/dev/peps/pep-0503/#normalized-names" rel="noopener nofollow ugc">normalizing rules from PEP 503</a>. It is crucial to us that the names are normalized, because on RPM level, <code>sectuptools-scm != sectuptools_scm</code>. By using the normalized form for both <em>Provides</em> and <em>Requires</em>, we don’t have this problem.</p> <p>We have recently started to generate such RPM <em>Provides</em> and <em>Requires</em> for extras as well, in a form of:</p> <pre><code>python3.Xdist(<normalized_package_name>[<extras_name>]) </code></pre> <p>E.g.</p> <pre><code>python3.9dist(sectuptools-scm[toml]) </code></pre> <p>Everything worked well <img src="https://emoji.discourse-cdn.com/apple/rainbow.png?v=9" title=":rainbow:" class="emoji" alt=":rainbow:"> <img src="https://emoji.discourse-cdn.com/apple/unicorn.png?v=9" title=":unicorn:" class="emoji" alt=":unicorn:"></p> <p>Until we started to deal with case sensitivity, underscores and such. It appears that pip treats <code>dnspython[DNSSEC]</code> and <code>dnspython[dnssec]</code> equally (good!), but it does not treat <code>webscrapbook[adhoc_ssl]</code> and <code>webscrapbook[adhoc-ssl]</code> equally <img src="https://emoji.discourse-cdn.com/apple/exploding_head.png?v=9" title=":exploding_head:" class="emoji" alt=":exploding_head:"></p> <p>What are the normalize rules for extras names and where are they defined? And if they are undefined, should they be defined? Would re-using the same normalizing function that exists for package names make sense?</p> <p>Thanks.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="6" /> <span class='post-likes'>6 Likes</span> </div> <div class='crawler-linkback-list' itemscope itemtype='http://schema.org/ItemList'> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://discuss.python.org/t/pep-685-comparison-of-extra-names-for-optional-distribution-dependencies/14141">PEP 685: Comparison of extra names for optional distribution dependencies</a> <meta itemprop='position' content='4'> </div> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://discuss.python.org/t/pep-685-comparison-of-extra-names-for-optional-distribution-dependencies/14141/58">PEP 685: Comparison of extra names for optional distribution dependencies</a> <meta itemprop='position' content='5'> </div> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://discuss.python.org/t/pep-685-comparison-of-extra-names-for-optional-distribution-dependencies/14141/59">PEP 685: Comparison of extra names for optional distribution dependencies</a> <meta itemprop='position' content='6'> </div> </div> </div> <div id='post_2' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/dustin'><span itemprop='name'>dustin</span></a> (Dustin Ingram) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-03-10T19:30:36Z' class='post-time'> March 10, 2021, 7:30pm </time> <meta itemprop='dateModified' content='2021-03-10T19:30:36Z'> <span itemprop='position'>2</span> </span> </div> <div class='post' itemprop='text'> <p>There is some specification at <a href="https://packaging.python.org/specifications/core-metadata/#provides-extra-multiple-use" class="inline-onebox" rel="noopener nofollow ugc">Core metadata specifications — Python Packaging User Guide</a> (important to note that an extra like <code>foo[bar, baz]</code> is legal) but normalization is not included there.</p> <p>This is probably an implicit standard via the implementation details of distutils/setuptools/pip that could stand to be explicitly defined.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_3' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/uranusjr'><span itemprop='name'>uranusjr</span></a> (Tzu-ping Chung) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-03-10T21:54:20Z' class='post-time'> March 10, 2021, 9:54pm </time> <meta itemprop='dateModified' content='2021-03-11T20:23:17Z'> <span itemprop='position'>3</span> </span> </div> <div class='post' itemprop='text'> <blockquote> <p>This is probably an implicit standard via the implementation details of distutils/setuptools/pip that could stand to be explicitly defined.</p> </blockquote> <p>And like many things in this area, it’s complicated <img src="https://emoji.discourse-cdn.com/apple/stuck_out_tongue.png?v=12" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:" loading="lazy" width="20" height="20"></p> <p>The de facto standard (<a href="https://github.com/pypa/setuptools/blob/b2f7b8f92725c63b164d5776f85e67cc560def4e/pkg_resources/__init__.py#L1324-L1330" rel="noopener nofollow ugc">implemented by setuptools</a>) is to convert all running characters outside <code>[A-Za-z0-9.-]</code> to a single underscore, and covert to lowercase. User can specify effectively anything in <code>setup.py</code>.</p> <p>But Core Metadata (linked above) says an extra name (quote) <em>Must be a valid Python identifier</em>—in other words, can’t contain <code>-</code> and <code>.</code>, and can’t start with a digit, etc.[1] This is in direct conflict to setuptools’s implementation, which would happily produce invalid extra names according to Core Metadata. (I’m not sure if <code>wheel</code> implements additional logic to block those names from being written into a wheel.)</p> <p>So if we’re going to have a standard on this, this would be a good opportunity to fix the available character set. I’m not sure what the best rule is here considering backwards compatibility. The general form is easy to agree on, but there are various edge cases like whether the name can begin with a digit or a non-alphanumeric character.</p> <p>The normalisation rule is probably good enough as-is, although I’d personally hope we can change to a rule similar to PEP 503 to avoid user confusion.</p> <p>[1]: Since the rule was written pre-Python 3, I assume non-ASCII characters are also out.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="3" /> <span class='post-likes'>3 Likes</span> </div> </div> <div id='post_4' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/hroncok'><span itemprop='name'>hroncok</span></a> (Miro Hrončok) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-03-16T14:26:19Z' class='post-time'> March 16, 2021, 2:26pm </time> <meta itemprop='dateModified' content='2021-03-16T17:35:50Z'> <span itemprop='position'>4</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="dustin" data-post="2" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/dustin/48/149_2.png" class="avatar"> dustin:</div> <blockquote> <p>important to note that an extra like <code>foo[bar, baz]</code> is legal</p> </blockquote> </aside> <p>Packaging (and hence our tooling) considers this as two extra names and we reduce the problem to <code>foo[bzr] && foo[baz]</code>.</p> <aside class="quote no-group" data-username="dustin" data-post="2" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/dustin/48/149_2.png" class="avatar"> dustin:</div> <blockquote> <p>This is probably an implicit standard via the implementation details of distutils/setuptools/pip that could stand to be explicitly defined.</p> </blockquote> </aside> <p>I’d happily help to define it explicitly.</p> <aside class="quote no-group" data-username="uranusjr" data-post="3" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/uranusjr/48/103_2.png" class="avatar"> uranusjr:</div> <blockquote> <p>The de facto standard (<a href="https://github.com/pypa/setuptools/blob/b2f7b8f92725c63b164d5776f85e67cc560def4e/pkg_resources/__init__.py#L1324-L1330" rel="noopener nofollow ugc">implemented by setuptools</a>) is to convert all running characters outside <code>[A-Za-z0-9.-]</code> to a single underscore, and covert to lowercase.</p> </blockquote> </aside> <p>That happens on “build time”. While definitively important, it is not what is determinative for this problem. I need to know what conversions (should) happen in installers when they resolve dependencies. Does pip follow the same logic?</p> <p>It parses the requirement via packaging:</p> <pre data-code-wrap="python"><code class="lang-python">ALPHANUM = Word(string.ascii_letters + string.digits) PUNCTUATION = Word("-_.") IDENTIFIER_END = ALPHANUM | (ZeroOrMore(PUNCTUATION) + ALPHANUM) IDENTIFIER = Combine(ALPHANUM + ZeroOrMore(IDENTIFIER_END)) EXTRA = IDENTIFIER </code></pre> <p>So it fails if it contains characters like <code>!</code> or <code>@</code>. Good.</p> <p>It seems that it does not consider any of the listed punctuation as equal.<br> It seems that it simply ignores the case.</p> <aside class="quote no-group" data-username="uranusjr" data-post="3" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/uranusjr/48/103_2.png" class="avatar"> uranusjr:</div> <blockquote> <p>The general form is easy to agree on, but there are various edge cases like whether the name can begin with a digit or a non-alphanumeric character.</p> </blockquote> </aside> <p>Let’s standardize what packaging already uses to parse it? It can begin with a digit, but not a punctuation.</p> <p>For 100% backwards compatibility, we could define the normalization rule as:</p> <pre data-code-wrap="python"><code class="lang-python">def normalize(extra_name): return extra_name.lower() </code></pre> <p>If we are not afraid of changes, we could define it to what PEP 503 does for names:</p> <pre data-code-wrap="python"><code class="lang-python">def normalize(extra_name): return re.sub(r"[-_.]+", "-", extra_name).lower() </code></pre> <p>That would require some changes in pip (or some of the vendored libraries). It would only blow up if some projects define multiple extras with names that only differ in punctuation. I’d say that is a no-risk, but maybe I am an optimist.</p> <p>Should I PEP this? The PEP would define the rules of “valid” extra names (defined by current packaging parser) and their normalized form (one of the above).</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="2" /> <span class='post-likes'>2 Likes</span> </div> </div> <div id='post_5' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/hroncok'><span itemprop='name'>hroncok</span></a> (Miro Hrončok) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-04-21T22:38:55Z' class='post-time'> April 21, 2021, 10:38pm </time> <meta itemprop='dateModified' content='2021-04-21T22:38:55Z'> <span itemprop='position'>5</span> </span> </div> <div class='post' itemprop='text'> <p>A gentle bump. I offer to work on a PEP, but I’d like to have a consensus first.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_6' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/uranusjr'><span itemprop='name'>uranusjr</span></a> (Tzu-ping Chung) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-04-25T08:50:23Z' class='post-time'> April 25, 2021, 8:50am </time> <meta itemprop='dateModified' content='2021-04-25T08:50:23Z'> <span itemprop='position'>6</span> </span> </div> <div class='post' itemprop='text'> <p>Sorry for the late reply (I was actually reminded by an entirely unrelated thread elsewhere that wants to add a uncommon extra name…)</p> <aside class="quote no-group" data-username="hroncok" data-post="4" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/hroncok/48/659_2.png" class="avatar"> hroncok:</div> <blockquote> <p>I need to know what conversions (should) happen in installers when they resolve dependencies. Does pip follow the same logic?</p> </blockquote> </aside> <p>Yes, pip normalises all extra names when they come in with the same <code>safe_extra()</code> function. So</p> <ol> <li>Only ASCII alphanumerics, <code>-</code>, <code>_</code>, and <code>.</code> are allowed in an extra name.</li> <li><code>-</code> and <code>.</code> are normalised to <code>_</code> for comparison. So <code>foo[my-bar]</code>, <code>foo[my_bar]</code>, and <code>foo[my.bar]</code> are equivalent.</li> <li>Case is folded.</li> </ol> <aside class="quote no-group" data-username="hroncok" data-post="4" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/hroncok/48/659_2.png" class="avatar"> hroncok:</div> <blockquote> <p>If we are not afraid of changes, we could define it to what PEP 503 does for names:</p> <pre><code class="lang-auto">def normalize(extra_name): return re.sub(r"[-_.]+", "-", extra_name).lower() </code></pre> </blockquote> </aside> <p>I think this would actually not cause any backwards compatibility issues for pip. Can’t say for other installers, but I don’t really care (they should’ve asked for a clarification like this thread before doing something different).</p> <aside class="quote no-group" data-username="hroncok" data-post="4" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/hroncok/48/659_2.png" class="avatar"> hroncok:</div> <blockquote> <p>Should I PEP this? The PEP would define the rules of “valid” extra names (defined by current packaging parser) and their normalized form (one of the above).</p> </blockquote> </aside> <p>Does this count as a “small specification change” described in <a href="https://discuss.python.org/t/pypa-governance-specification-updates/8134">PyPA Governance - Specification Updates</a>? I’d say let’s first send a clarification PR to <a href="https://packaging.python.org/specifications/" rel="noopener nofollow ugc">PyPA specifications</a> first, and write a PEP only if someone thinks it is required. <img src="https://emoji.discourse-cdn.com/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_7' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/pf_moore'><span itemprop='name'>pf_moore</span></a> (Paul Moore) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-04-25T09:31:08Z' class='post-time'> April 25, 2021, 9:31am </time> <meta itemprop='dateModified' content='2021-04-25T09:31:08Z'> <span itemprop='position'>7</span> </span> </div> <div class='post' itemprop='text'> <p>From the <a href="https://www.pypa.io/en/latest/specifications/#handling-fixes-and-other-minor-updates">rules referenced in that thread</a>:</p> <blockquote> <p>If a change being considered this way has the potential to affect software interoperability, then it must be escalated to the distutils-sig mailing list for discussion, where it will be either approved as a text-only change, or else directed to the PEP process for specification updates.</p> </blockquote> <p>It’s not entirely clear what constitutes “approval” here, but I’d take the view that getting a consensus is sufficient, with the PEP-delegate having a veto (not that the latter is relevant in this case - see next sentence <img src="https://emoji.discourse-cdn.com/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20">).</p> <p>I’m fine with doing this as a spec update, as long as no-one else wants to argue that it needs to be a PEP.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_8' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/hroncok'><span itemprop='name'>hroncok</span></a> (Miro Hrončok) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-04-25T20:45:40Z' class='post-time'> April 25, 2021, 8:45pm </time> <meta itemprop='dateModified' content='2021-04-25T20:45:40Z'> <span itemprop='position'>8</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="uranusjr" data-post="6" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/uranusjr/48/103_2.png" class="avatar"> uranusjr:</div> <blockquote> <p><code>-</code> and <code>.</code> are normalised to <code>_</code> for comparison. So <code>foo[my-bar]</code> , <code>foo[my_bar]</code> , and <code>foo[my.bar]</code> are equivalent.</p> </blockquote> </aside> <p>If that is the <em>intended</em> behavior, there must be a bug somewhere:</p> <pre><code class="lang-auto">(venv) [tmp]$ pip install -U pip ... Successfully installed pip-21.1 (venv) [tmp]$ pip install 'webscrapbook[adhoc-ssl]' Collecting webscrapbook[adhoc-ssl] Downloading webscrapbook-0.40.0-py3-none-any.whl (146 kB) |████████████████████████████████| 146 kB 2.1 MB/s WARNING: webscrapbook 0.40.0 does not provide the extra 'adhoc-ssl' ... Installing collected packages: MarkupSafe, werkzeug, jinja2, itsdangerous, click, lxml, flask, commonmark, webscrapbook Successfully installed MarkupSafe-1.1.1 click-7.1.2 commonmark-0.9.1 flask-1.1.2 itsdangerous-1.1.0 jinja2-2.11.3 lxml-4.6.3 webscrapbook-0.40.0 werkzeug-1.0.1 (venv) [tmp]$ pip install 'webscrapbook[adhoc_ssl]' ... Installing collected packages: pycparser, cffi, cryptography Successfully installed cffi-1.14.5 cryptography-3.4.7 pycparser-2.20 </code></pre> <p>That is why I started this thread.</p> <aside class="quote group-committers" data-username="pf_moore" data-post="7" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/pf_moore/48/35_2.png" class="avatar"> pf_moore:</div> <blockquote> <p>I’m fine with doing this as a spec update, as long as no-one else wants to argue that it needs to be a PEP.</p> </blockquote> </aside> <p>Works for me! I’ve only said PEP because I assumed it was required.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_9' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/pf_moore'><span itemprop='name'>pf_moore</span></a> (Paul Moore) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-04-25T21:49:33Z' class='post-time'> April 25, 2021, 9:49pm </time> <meta itemprop='dateModified' content='2021-04-25T21:49:33Z'> <span itemprop='position'>9</span> </span> </div> <div class='post' itemprop='text'> <p>Note that I said it needs “consensus” - I’d like to see a few more people agree that the proposed behaviour is acceptable here before it goes to a PR. The key is to give interested parties a chance to see the proposal and comment in a well-known forum, and the tracker for a PR doesn’t qualify for that IMO.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_10' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/uranusjr'><span itemprop='name'>uranusjr</span></a> (Tzu-ping Chung) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-04-26T06:41:10Z' class='post-time'> April 26, 2021, 6:41am </time> <meta itemprop='dateModified' content='2021-04-26T06:41:10Z'> <span itemprop='position'>10</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="hroncok" data-post="8" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/hroncok/48/659_2.png" class="avatar"> hroncok:</div> <blockquote> <p>If that is the <em>intended</em> behavior, there must be a bug somewhere:</p> </blockquote> </aside> <p>Indeed! I checked again and pip’s extra normalisaton behaviour is quite convoluted and eratic. There is code to normalise extras, but I couldn’t find anywhere the logic is ever reached, and can’t help but wonder maybe this worked at some point in the past and silently broke without anyone noticing. I guess that’s yet another reason to have an enforced specification around this…</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="3" /> <span class='post-likes'>3 Likes</span> </div> </div> <div id='post_11' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/hroncok'><span itemprop='name'>hroncok</span></a> (Miro Hrončok) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-12-20T19:30:50Z' class='post-time'> December 20, 2021, 7:30pm </time> <meta itemprop='dateModified' content='2021-12-20T19:30:50Z'> <span itemprop='position'>11</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-committers" data-username="pf_moore" data-post="9" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/pf_moore/48/35_2.png" class="avatar"> pf_moore:</div> <blockquote> <p>I’d like to see a few more people agree that the proposed behaviour is acceptable here before it goes to a PR.</p> </blockquote> </aside> <p>There has been no disagreement here, but neither there was agreement. Is there any place I need to go to and promote this discussion there?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_12' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/pf_moore'><span itemprop='name'>pf_moore</span></a> (Paul Moore) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-12-20T19:53:44Z' class='post-time'> December 20, 2021, 7:53pm </time> <meta itemprop='dateModified' content='2021-12-20T19:53:44Z'> <span itemprop='position'>12</span> </span> </div> <div class='post' itemprop='text'> <p>I don’t know, to be honest. Maybe it needs a PEP just to raise visibility and trigger a proper discussion. At a minumum, given that setuptools and pip are mentioned in the thread, I think you need to get explicit confirmation from the maintainers of those projects that they don’t have a problem with whatever you are proposing. <a class="mention" href="/u/uranusjr">@uranusjr</a> has commented here already, so I guess that’s sufficient for pip (I’m also a pip maintainer but I haven’t researched the question and how it would affect pip).</p> <p>If you can’t get sufficient voices confirming it can be done as a spec update, the fallback is that it’s a PEP.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_13' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/uranusjr'><span itemprop='name'>uranusjr</span></a> (Tzu-ping Chung) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-12-21T01:47:02Z' class='post-time'> December 21, 2021, 1:47am </time> <meta itemprop='dateModified' content='2021-12-21T01:47:02Z'> <span itemprop='position'>13</span> </span> </div> <div class='post' itemprop='text'> <p>I think a PEP is needed, but have had not enough time to write it yet. The specification should be fairly short, but is substential enough to be its own document IMO. It needs to standardise at least two things:</p> <ol> <li> <p>What extra names should be considered equivalent. This should use <code>pkg_resources.safe_extra()</code> since it’s the only logic that can possibly work without breaking things.</p> </li> <li> <p>How extras should be compared in a PEP 508 environment marker. This is needed because PEP 508 does not sufficiently define how the value of <code>extra</code> should be compared. There are two possibilities:</p> <ol> <li>The standard should mandate all metadata-producing tools to normalise an extra before putting it into a marker (i.e. should write e.g. <code>foo; extra == 'x.y'</code> instead of <code>foo; extra == 'X.Y'</code>).</li> <li>Marker evaluation logic (e.g. <code>packaging.marker</code>) must be amended to perform normalisation when comparing markers (i.e. <code>Marker("extra == 'X.Y'").evaluate({'extra': 'x.y'})</code> must return <code>True</code>).</li> </ol> <p>I think the latter solution is likely more viable, since we can’t fix metadata in existing packages on PyPI.</p> </li> </ol> <p>This topic actually came up again when I was working on pip’s <code>importlib.metadata</code> support. It’s really a PITA that I wish more people can take interest in.</p> <aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pypa/pip/pull/10709/files#diff-dc6d0ea33ced90d3436f4edb226bbccebdddf1bdf8a0c91b2cc178546198769eR231-R256"> <header class="source"> <a href="https://github.com/pypa/pip/pull/10709/files#diff-dc6d0ea33ced90d3436f4edb226bbccebdddf1bdf8a0c91b2cc178546198769eR231-R256" target="_blank" rel="noopener nofollow ugc">github.com/pypa/pip</a> </header> <article class="onebox-body"> <div class="github-row"> <div class="github-icon-container" title="Pull Request"> <svg width="60" height="60" class="github-icon" viewbox="0 0 12 16" aria-hidden="true"><path d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg> </div> <div class="github-info-container"> <h4> <a href="https://github.com/pypa/pip/pull/10709" target="_blank" rel="noopener nofollow ugc">Metadata backend with importlib.metadata</a> </h4> <div class="branches"> <code>pypa:main</code> ← <code>uranusjr:metadata-importlib-backend</code> </div> <div class="github-info"> <div class="date"> opened <span class="discourse-local-date" data-format="ll" data-date="2021-12-05" data-time="00:42:56" data-timezone="UTC">12:42AM - 05 Dec 21 UTC</span> </div> <div class="user"> <a href="https://github.com/uranusjr" target="_blank" rel="noopener nofollow ugc"> <img alt="uranusjr" src="https://us1.discourse-cdn.com/flex002/uploads/python1/original/2X/6/66ff25ed3714071050896a44e6da57bdfc2998d3.png" class="onebox-avatar-inline" width="20" height="20"> uranusjr </a> </div> <div class="lines" title="22 commits changed 14 files with 637 additions and 82 deletions"> <a href="https://github.com/pypa/pip/pull/10709/files" target="_blank" rel="noopener nofollow ugc"> <span class="added">+637</span> <span class="removed">-82</span> </a> </div> </div> </div> </div> <div class="github-row"> <p class="github-body-container">This doesn’t take a lot of code because we’ve done most work upfront. There are <span class="show-more-container"><a href="https://github.com/pypa/pip/pull/10709" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">definitely optimisations available since importlib.metadata has a pretty different implementation, but this should make the most common cases “work”. Likely far from perfect, I’m only submitting this as draft to run a new workflow against it to see how things currently stand. Currently the backend is only accessible via a private environment variable. Eventually I want to target the implementation to only 3.10 or later (because importlib.metadata API is quite unstable before that), expose behind the feature flag first, and slowly rolling it out to maybe 3.11 the earliest.</span></p> </div> </article> <div class="onebox-metadata"> </div> <div style="clear: both"></div> </aside> <p>(See <code>_iter_egg_info_dependencies</code> in <code>pip/_internal/metadata/importlib/_dists.py</code>.)</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_14' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/pf_moore'><span itemprop='name'>pf_moore</span></a> (Paul Moore) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-12-21T08:25:47Z' class='post-time'> December 21, 2021, 8:25am </time> <meta itemprop='dateModified' content='2021-12-21T08:25:47Z'> <span itemprop='position'>14</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="uranusjr" data-post="13" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/uranusjr/48/103_2.png" class="avatar"> uranusjr:</div> <blockquote> <p>I think the latter solution is likely more viable, since we can’t fix metadata in existing packages on PyPI.</p> </blockquote> </aside> <p>I think option (1) is better (we recently had a similar debate about whether project names should be explicitly normalised, which I think we should avoid repeating) but we should also require (2) for compatibility with existing un-normalised data. (That’s the “Transition plan” section I’m proposing we include in new PEPs <img src="https://emoji.discourse-cdn.com/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20">)</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_15' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/brettcannon'><span itemprop='name'>brettcannon</span></a> (Brett Cannon) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-12-21T19:11:30Z' class='post-time'> December 21, 2021, 7:11pm </time> <meta itemprop='dateModified' content='2021-12-21T22:55:17Z'> <span itemprop='position'>15</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="uranusjr" data-post="13" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/uranusjr/48/103_2.png" class="avatar"> uranusjr:</div> <blockquote> <p>What extra names should be considered equivalent. This should use <code>pkg_resources.safe_extra()</code></p> </blockquote> </aside> <p>For anyone else who, like me, doesn’t know what that involves:</p> <p><a href="https://github.com/pypa/setuptools/blob/390016f98244e8c11d6060c3aa27efbc36db8592/pkg_resources/__init__.py#L1329-L1335">GitHub link</a></p> <aside class="quote no-group" data-username="uranusjr" data-post="13" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/uranusjr/48/103_2.png" class="avatar"> uranusjr:</div> <blockquote> <ul> <li>The standard should mandate all metadata-producing tools to normalise an extra before putting it into a marker (i.e. should write e.g. <code>foo; extra == 'x.y'</code> instead of <code>foo; extra == 'X.Y'</code> ).</li> <li>Marker evaluation logic (e.g. <code>packaging.marker</code> ) must be amended to perform normalisation when comparing markers (i.e. <code>Marker("extra == 'X.Y'").evaluate({'extra': 'x.y'})</code> must return <code>True</code> ).</li> </ul> </blockquote> </aside> <p>Seems reasonable: loose on the input, strict on the output.</p> <aside class="quote no-group" data-username="uranusjr" data-post="13" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/uranusjr/48/103_2.png" class="avatar"> uranusjr:</div> <blockquote> <p>This topic actually came up again when I was working on pip’s <code>importlib.metadata</code> support. It’s really a PITA that I wish more people can take interest in.</p> </blockquote> </aside> <p>Is it just how to normalize extras, or is there something bigger you’re referring to?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_16' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/njs'><span itemprop='name'>njs</span></a> (Nathaniel J. Smith) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-12-21T23:01:17Z' class='post-time'> December 21, 2021, 11:01pm </time> <meta itemprop='dateModified' content='2021-12-21T23:01:17Z'> <span itemprop='position'>16</span> </span> </div> <div class='post' itemprop='text'> <p>Can we make extra validation/normalization the same as package name validation/normalization? It looks like it’s pretty similar, at least. That would be useful for simplicity, and also to keep our options open for reifying extras as part of the package name in the future.</p> <p>For marker evaluation: simplest might be to declare that <code>extra</code> can only appear on the left-hand side of a <code>==</code> or <code>!=</code>, and that this then uses normalizing comparison rules?</p> <p>If we do that then I don’t really see the point of also mandating that tools produce a specific string form.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_17' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/uranusjr'><span itemprop='name'>uranusjr</span></a> (Tzu-ping Chung) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-12-22T08:18:59Z' class='post-time'> December 22, 2021, 8:18am </time> <meta itemprop='dateModified' content='2021-12-22T08:18:59Z'> <span itemprop='position'>17</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-committers" data-username="njs" data-post="16" data-topic="7614"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/njs/48/204_2.png" class="avatar"> njs:</div> <blockquote> <p>Can we make extra validation/normalization the same as package name validation/normalization?</p> </blockquote> </aside> <p>It’s theoratically possible, but I’m not sure if it’s a good idea to declare all existing package managers broken to persue theoratical purity. You’d get no objection from me if you write that into a PEP, but I’m not going to try writing that PEP myself and defending the decision against user complaints.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_18' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/njs'><span itemprop='name'>njs</span></a> (Nathaniel J. Smith) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-12-22T09:54:50Z' class='post-time'> December 22, 2021, 9:54am </time> <meta itemprop='dateModified' content='2021-12-22T09:54:50Z'> <span itemprop='position'>18</span> </span> </div> <div class='post' itemprop='text'> <p>Can you give some examples of extra names that would be broken? From a quick look it seems like <code>safe_extra</code> and PEP 503 normalization are equivalent but I could easily be missing some details.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_19' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/pf_moore'><span itemprop='name'>pf_moore</span></a> (Paul Moore) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-12-22T10:07:25Z' class='post-time'> December 22, 2021, 10:07am </time> <meta itemprop='dateModified' content='2021-12-22T10:07:25Z'> <span itemprop='position'>19</span> </span> </div> <div class='post' itemprop='text'> <p><code>a£b</code>. Also, safe_extra uses <code>_</code> as the replacement character whereas PEP 503 uses <code>-</code>.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_20' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/njs'><span itemprop='name'>njs</span></a> (Nathaniel J. Smith) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2021-12-22T10:39:47Z' class='post-time'> December 22, 2021, 10:39am </time> <meta itemprop='dateModified' content='2021-12-22T10:39:47Z'> <span itemprop='position'>20</span> </span> </div> <div class='post' itemprop='text'> <p>I don’t think the replacement character matters, since users are always supposed to re-normalize before doing any comparisons. So if pip or whatever wants to prefer one replacement character or another internally, it doesn’t affect anything.</p> <p>I guess the difference in <code>£</code> is that <code>safe_extra</code> treats it as punctuation, <code>a£b == a-b</code>, while PEP 503 says that it’s illegal? And same for every other character that’s not ASCII alphanumerics, <code>-</code>, <code>.</code> or <code>_</code>?</p> <p>The <code>safe_extra</code> approach doesn’t seem very useful – "sure, you can write your extra name in greek or cyrillic, but all extras written in those alphabets will be interpreted as if you had written a single <code>"_"</code>". And making those characters illegal probably wouldn’t be <em>too</em> disruptive – I doubt many people are using them? But idk if it’s like, “literally no-one” or “1 package” or “100 packages”, so maybe it would be disruptive enough to not be worth it, not sure.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> </div> <div role='navigation' itemscope itemtype='http://schema.org/SiteNavigationElement' class="topic-body crawler-post"> <span itemprop='name'><b><a rel="next" itemprop="url" href="/t/what-extras-names-are-treated-as-equal-and-why/7614?page=2">next page →</a></b></span> </div> </div> <footer class="container wrap"> <nav class='crawler-nav'> <ul> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/' itemprop="url">Home </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/categories' itemprop="url">Categories </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/guidelines' itemprop="url">Guidelines </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/tos' itemprop="url">Terms of Service </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/privacy' itemprop="url">Privacy Policy </a> </span> </li> </ul> </nav> <p class='powered-by-link'>Powered by <a href="https://www.discourse.org">Discourse</a>, best viewed with JavaScript enabled</p> </footer> <div class="buorg"><div>Unfortunately, <a href="https://www.discourse.org/faq/#browser">your browser is unsupported</a>. Please <a href="https://browsehappy.com">switch to a supported browser</a> to view rich content, log in and reply.</div></div> </body> </html>