Community | Apache Spark

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title> Community | Apache Spark </title> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous"> <link rel="preconnect" href="https://fonts.googleapis.com"> <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> <link href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,wght@0,400;0,500;0,700;1,400;1,500;1,700&Courier+Prime:wght@400;700&display=swap" rel="stylesheet"> <link href="/css/custom.css" rel="stylesheet">  <link href="/css/pygments-default.css" rel="stylesheet"> <link rel="icon" href="/favicon.ico" type="image/x-icon">  <script> var _paq = window._paq = window._paq || []; /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ _paq.push(["disableCookies"]); _paq.push(['trackPageView']); _paq.push(['enableLinkTracking']); (function() { var u="https://analytics.apache.org/"; _paq.push(['setTrackerUrl', u+'matomo.php']); _paq.push(['setSiteId', '40']); var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); })(); </script>  </head> <body class="global"> <nav class="navbar navbar-expand-lg navbar-dark p-0 px-4" style="background: #1D6890;"> <a class="navbar-brand" href="/"> <img src="/images/spark-logo-rev.svg" alt="" width="141" height="72"> </a> <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarContent" aria-controls="navbarContent" aria-expanded="false" aria-label="Toggle navigation"> <span class="navbar-toggler-icon"></span> </button> <div class="collapse navbar-collapse col-md-12 col-lg-auto pt-4" id="navbarContent"> <ul class="navbar-nav me-auto"> <li class="nav-item"> <a class="nav-link active" aria-current="page" href="/downloads.html">Download</a> </li> <li class="nav-item dropdown"> <a class="nav-link dropdown-toggle" href="#" id="libraries" role="button" data-bs-toggle="dropdown" aria-expanded="false"> Libraries </a> <ul class="dropdown-menu" aria-labelledby="libraries"> <li><a class="dropdown-item" href="/sql/">SQL and DataFrames</a></li> <li><a class="dropdown-item" href="/spark-connect/">Spark Connect</a></li> <li><a class="dropdown-item" href="/streaming/">Spark Streaming</a></li> <li><a class="dropdown-item" href="/pandas-on-spark/">pandas on Spark</a></li> <li><a class="dropdown-item" href="/mllib/">MLlib (machine learning)</a></li> <li><a class="dropdown-item" href="/graphx/">GraphX (graph)</a></li> <li> <hr class="dropdown-divider"> </li> <li><a class="dropdown-item" href="/third-party-projects.html">Third-Party Projects</a></li> </ul> </li> <li class="nav-item dropdown"> <a class="nav-link dropdown-toggle" href="#" id="documentation" role="button" data-bs-toggle="dropdown" aria-expanded="false"> Documentation </a> <ul class="dropdown-menu" aria-labelledby="documentation"> <li><a class="dropdown-item" href="/docs/latest/">Latest Release</a></li> <li><a class="dropdown-item" href="/documentation.html">Older Versions and Other Resources</a></li> <li><a class="dropdown-item" href="/faq.html">Frequently Asked Questions</a></li> </ul> </li> <li class="nav-item"> <a class="nav-link active" aria-current="page" href="/examples.html">Examples</a> </li> <li class="nav-item dropdown"> <a class="nav-link dropdown-toggle" href="#" id="community" role="button" data-bs-toggle="dropdown" aria-expanded="false"> Community </a> <ul class="dropdown-menu" aria-labelledby="community"> <li><a class="dropdown-item" href="/community.html">Mailing Lists & Resources</a></li> <li><a class="dropdown-item" href="/contributing.html">Contributing to Spark</a></li> <li><a class="dropdown-item" href="/improvement-proposals.html">Improvement Proposals (SPIP)</a> </li> <li><a class="dropdown-item" href="https://issues.apache.org/jira/browse/SPARK">Issue Tracker</a> </li> <li><a class="dropdown-item" href="/powered-by.html">Powered By</a></li> <li><a class="dropdown-item" href="/committers.html">Project Committers</a></li> <li><a class="dropdown-item" href="/history.html">Project History</a></li> </ul> </li> <li class="nav-item dropdown"> <a class="nav-link dropdown-toggle" href="#" id="developers" role="button" data-bs-toggle="dropdown" aria-expanded="false"> Developers </a> <ul class="dropdown-menu" aria-labelledby="developers"> <li><a class="dropdown-item" href="/developer-tools.html">Useful Developer Tools</a></li> <li><a class="dropdown-item" href="/versioning-policy.html">Versioning Policy</a></li> <li><a class="dropdown-item" href="/release-process.html">Release Process</a></li> <li><a class="dropdown-item" href="/security.html">Security</a></li> </ul> </li> <li class="nav-item dropdown"> <a class="nav-link dropdown-toggle" href="#" id="github" role="button" data-bs-toggle="dropdown" aria-expanded="false"> GitHub </a> <ul class="dropdown-menu" aria-labelledby="github"> <li><a class="dropdown-item" href="https://github.com/apache/spark">spark</a></li> <li><a class="dropdown-item" href="https://github.com/apache/spark-connect-go">spark-connect-go</a></li> <li><a class="dropdown-item" href="https://github.com/apache/spark-docker">spark-docker</a></li> <li><a class="dropdown-item" href="https://github.com/apache/spark-kubernetes-operator">spark-kubernetes-operator</a></li> <li><a class="dropdown-item" href="https://github.com/apache/spark-website">spark-website</a></li> </ul> </li> </ul> <ul class="navbar-nav ml-auto"> <li class="nav-item dropdown"> <a class="nav-link dropdown-toggle" href="#" id="apacheFoundation" role="button" data-bs-toggle="dropdown" aria-expanded="false"> Apache Software Foundation </a> <ul class="dropdown-menu" aria-labelledby="apacheFoundation"> <li><a class="dropdown-item" href="https://www.apache.org/">Apache Homepage</a></li> <li><a class="dropdown-item" href="https://www.apache.org/licenses/">License</a></li> <li><a class="dropdown-item" href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> <li><a class="dropdown-item" href="https://www.apache.org/foundation/thanks.html">Thanks</a></li> <li><a class="dropdown-item" href="https://www.apache.org/security/">Security</a></li> <li><a class="dropdown-item" href="https://www.apache.org/events/current-event">Event</a></li> </ul> </li> </ul> </div> </nav> <div class="container"> <div class="row mt-4"> <div class="col-12 col-md-9"> <h2>Apache Spark<span class="tm">™</span> community</h2> <p><a name="have-questions"></a></p> <h3>Have questions?</h3> <h4>StackOverflow</h4> <p>For usage questions and help (e.g. how to use this Spark API), it is recommended you use the StackOverflow tag <a href="https://stackoverflow.com/questions/tagged/apache-spark"><code class="language-plaintext highlighter-rouge">apache-spark</code></a> as it is an active forum for Spark users’ questions and answers.</p> <p>Some quick tips when using StackOverflow:</p> <ul> <li>Prior to asking submitting questions, please: <ul> <li>Search StackOverflow’s <a href="https://stackoverflow.com/questions/tagged/apache-spark"><code class="language-plaintext highlighter-rouge">apache-spark</code></a> tag to see if your question has already been answered</li> <li>Search the ASF archive for <a href="https://lists.apache.org/list.html?user@spark.apache.org">user@spark.apache.org</a></li> </ul> </li> <li>Please follow the StackOverflow <a href="https://stackoverflow.com/help/how-to-ask">code of conduct</a></li> <li>Always use the <code class="language-plaintext highlighter-rouge">apache-spark</code> tag when asking questions</li> <li>Please also use a secondary tag to specify components so subject matter experts can more easily find them. Examples include: <code class="language-plaintext highlighter-rouge">pyspark</code>, <code class="language-plaintext highlighter-rouge">spark-dataframe</code>, <code class="language-plaintext highlighter-rouge">spark-streaming</code>, <code class="language-plaintext highlighter-rouge">spark-r</code>, <code class="language-plaintext highlighter-rouge">spark-mllib</code>, <code class="language-plaintext highlighter-rouge">spark-ml</code>, <code class="language-plaintext highlighter-rouge">spark-graphx</code>, <code class="language-plaintext highlighter-rouge">spark-graphframes</code>, <code class="language-plaintext highlighter-rouge">spark-tensorframes</code>, etc.</li> <li>Please do not cross-post between StackOverflow and the mailing lists</li> <li>No jobs, sales, or solicitation is permitted on StackOverflow</li> </ul> <p><a name="mailing-lists"></a></p> <h4>Mailing lists</h4> <p>For broad, opinion based, ask for external resources, debug issues, bugs, contributing to the project, and scenarios, it is recommended you use the user@spark.apache.org mailing list.</p> <ul> <li> <a href="https://lists.apache.org/list.html?user@spark.apache.org">user@spark.apache.org</a> is for usage questions, help, and announcements. <a href="mailto:user-subscribe@spark.apache.org?subject=(send%20this%20email%20to%20subscribe)">(subscribe)</a> <a href="mailto:user-unsubscribe@spark.apache.org?subject=(send%20this%20email%20to%20unsubscribe)">(unsubscribe)</a> <a href="https://lists.apache.org/list.html?user@spark.apache.org">(archives)</a> </li> <li> <a href="https://lists.apache.org/list.html?dev@spark.apache.org">dev@spark.apache.org</a> is for people who want to contribute code to Spark. <a href="mailto:dev-subscribe@spark.apache.org?subject=(send%20this%20email%20to%20subscribe)">(subscribe)</a> <a href="mailto:dev-unsubscribe@spark.apache.org?subject=(send%20this%20email%20to%20unsubscribe)">(unsubscribe)</a> <a href="https://lists.apache.org/list.html?dev@spark.apache.org">(archives)</a> </li> </ul> <p>Some quick tips when using email:</p> <ul> <li>Prior to asking submitting questions, please: <ul> <li>Search StackOverflow at <a href="https://stackoverflow.com/questions/tagged/apache-spark"><code class="language-plaintext highlighter-rouge">apache-spark</code></a> to see if your question has already been answered</li> <li>Search the ASF archive for <a href="https://lists.apache.org/list.html?user@spark.apache.org">user@spark.apache.org</a></li> </ul> </li> <li>Tagging the subject line of your email will help you get a faster response, e.g. <code class="language-plaintext highlighter-rouge">[Spark SQL]: Does Spark SQL support LEFT SEMI JOIN?</code></li> <li>Tags may help identify a topic by: <ul> <li>Component: Spark Core, Spark SQL, ML, MLlib, GraphFrames, GraphX, TensorFrames, etc</li> <li>Level: Beginner, Intermediate, Advanced</li> <li>Scenario: Debug, How-to</li> </ul> </li> <li>For error logs or long code examples, please use <a href="https://gist.github.com/">GitHub gist</a> and include only a few lines of the pertinent code / log within the email.</li> <li>No jobs, sales, or solicitation is permitted on the Apache Spark mailing lists.</li> </ul> <h5>Reporting security issues</h5> <p>Please see the <a href="security.html">Security</a> page for information on how to report sensitive security vulnerabilities, and for information on known security issues.</p> <p><a name="chat"></a></p> <h4>Chat rooms</h4> <p>Chat rooms are great for quick questions or discussions on specialized topics. The following chat rooms are not officially part of Apache Spark; they are provided for reference only.</p> <ul> <li> <a href="https://gitter.im/spark-scala/Lobby">Spark with Scala</a> is for questions and discussions related to using Spark with the Scala programming language. </li> </ul> <p><a name="events"></a></p> <h3>Events and meetups</h3> <h4>Conferences</h4> <ul> <li> <a href="https://spark-summit.org/eu-2017/">Spark Summit Europe 2017.</a> Oct 24 - 26 in Dublin. </li> <li> <a href="https://spark-summit.org/2017/">Spark Summit 2017.</a> June 5 - 7 in San Francisco. </li> <li> <a href="https://spark-summit.org/eu-2016/">Spark Summit Europe 2016.</a> Oct 25 - 27 in Brussels. </li> <li> <a href="https://spark-summit.org/2016/">Spark Summit 2016.</a> June 6 - 8 in San Francisco. </li> <li> <a href="https://spark-summit.org/east-2016/">Spark Summit East 2016.</a> Feb 16 - 18 in New York City. </li> <li> <a href="https://spark-summit.org/eu-2015/">Spark Summit Europe 2015.</a> Oct 27 - 29 in Amsterdam. </li> <li> <a href="https://spark-summit.org/2015">Spark Summit 2015.</a> June 15 - 17 in San Francisco. </li> <li> <a href="https://spark-summit.org/east-2015/">Spark Summit East 2015.</a> March 18 - 19 in New York City. </li> <li> <a href="https://spark-summit.org/2014">Spark Summit 2014.</a> June 30 - July 1 2014 in San Francisco. </li> <li> <a href="https://spark-summit.org/2013">Spark Summit 2013.</a> December 2013 in San Francisco. </li> </ul> <h4>Meetups</h4> <p>Spark Meetups are grass-roots events organized and hosted by individuals in the community around the world. Check out <a href="https://www.meetup.com/topics/apache-spark/">meetup.com/topics/apache-spark</a> to find a Spark meetup in your part of the world. Below is a partial list of Spark meetups.</p> <ul> <li> <a href="https://www.meetup.com/spark-users/">Bay Area Spark Meetup.</a> This group has been running since January 2012 in the San Francisco area. The meetup page also contains an <a href="https://www.meetup.com/spark-users/events/past/">archive of past meetups</a>, including videos and <a href="https://www.meetup.com/spark-users/files/">slides</a> for most of the recent talks. </li> <li> <a href="https://www.meetup.com/Spark-Barcelona/">Barcelona Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/SanKir-Big-Data-Group/">Bangalore Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/es/Apache-Spark-Bogota/">Bogot谩 Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Boston-Apache-Spark-User-Group/">Boston Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Boulder-Denver-Spark-Meetup/">Boulder/Denver Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Christchurch-Apache-Spark-Meetup/">Christchurch Apache Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Hangzhou-Apache-Spark-Meetup/">Hangzhou Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/israel-spark-users/">Israel Spark Users</a> </li> <li> <a href="https://www.meetup.com/Apache-Spark-Ljubljana-Meetup/">Ljubljana Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Spark-London/">London Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Apache-Spark-Maryland/">Maryland Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/es/apache-spark-mexicocity/">M茅xico City Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Apache-Spark-in-Moscow/">Moscow Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Mumbai-Spark-Meetup/">Mumbai Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Spark-NYC/">NYC Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Seattle-Spark-Meetup/">Seattle Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Shanghai-Apache-Spark-Meetup/">Shanghai Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Shenzhen-Apache-Spark-Meetup/">Shenzhen Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Tokyo-Spark-Meetup/">Tokyo Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Vancouver-Spark/">Vancouver Spark Meetup</a> </li> <li> <a href="https://www.meetup.com/Washington-DC-Area-Spark-Interactive/">Washington DC Area Spark Meetup</a> </li> </ul> <p>If you'd like your meetup or conference added, please email <a href="mailto:user@spark.apache.org">user@spark.apache.org</a>.</p> <p><a name="issue-tracker"></a></p> <h3>Issue tracker</h3> <p>The project tracks bugs and new features on <a href="https://issues.apache.org/jira/browse/SPARK">JIRA</a>. If you'd like, you can also subscribe to <a href="https://mail-archives.apache.org/mod_mbox/spark-issues/">issues@spark.apache.org</a> to receive emails about new issues, and <a href="https://mail-archives.apache.org/mod_mbox/spark-commits/">commits@spark.apache.org</a> to get emails about commits.</p> <h3>Shop</h3> <p><img alt="RedBubble" src="/images/redbubble.png" title="RedBubble" /></p> <p>The ASF has an <a href="https://www.redbubble.com/people/comdev/shop">official store at RedBubble</a> that Apache Community Development (ComDev) runs. In the store, <a href="https://www.redbubble.com/shop/ap/113203780">various products featuring the Apache Spark logo</a> are available.</p> <h3>Powered by</h3> <p>Our site has a list of <a href="/powered-by.html">projects and organizations powered by Spark</a>. Add yours by emailing `dev@spark.apache.org`. </p> </div> <div class="col-12 col-md-3"> <div class="news" style="margin-bottom: 20px;"> <h5>Latest News</h5> <ul class="list-unstyled"> <li><a href="/news/spark-3-4-4-released.html">Spark 3.4.4 released</a> <span class="small">(Oct 27, 2024)</span></li> <li><a href="/news/spark-4.0.0-preview2.html">Preview release of Spark 4.0</a> <span class="small">(Sep 26, 2024)</span></li> <li><a href="/news/spark-3-5-3-released.html">Spark 3.5.3 released</a> <span class="small">(Sep 24, 2024)</span></li> <li><a href="/news/spark-3-5-2-released.html">Spark 3.5.2 released</a> <span class="small">(Aug 10, 2024)</span></li> </ul> <p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p> </div> <div style="text-align:center; margin-bottom: 20px;"> <a href="https://www.apache.org/events/current-event.html"> <img src="https://www.apache.org/events/current-event-234x60.png" style="max-width: 100%;"/> </a> </div> <div class="hidden-xs hidden-sm"> <a href="/downloads.html" class="btn btn-cta btn-lg d-grid" style="margin-bottom: 30px;"> Download Spark </a> <p style="font-size: 16px; font-weight: 500; color: #555;"> Built-in Libraries: </p> <ul class="list-none"> <li><a href="/sql/">SQL and DataFrames</a></li> <li><a href="/streaming/">Spark Streaming</a></li> <li><a href="/mllib/">MLlib (machine learning)</a></li> <li><a href="/graphx/">GraphX (graph)</a></li> </ul> <a href="/third-party-projects.html">Third-Party Projects</a> </div> </div> </div> <footer class="small"> <hr> Apache Spark, Spark, Apache, the Apache feather logo, and the Apache Spark project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. See guidance on use of Apache Spark <a href="/trademarks.html">trademarks</a>. All other marks mentioned may be trademarks or registered trademarks of their respective owners. Copyright © 2018 The Apache Software Foundation, Licensed under the <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>. </footer> </div> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" crossorigin="anonymous"></script> <script src="https://code.jquery.com/jquery.js"></script> <script src="/js/lang-tabs.js"></script> <script src="/js/downloads.js"></script> </body> </html>

CINXE.COM

Community | Apache Spark