Fluentd Blog

<?xml version="1.0" encoding="utf-8"?> <rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/"> <channel> <title>Fluentd Blog</title> <link>https://www.fluentd.org/blog</link> <description>Get the Latest Updates on Fluentd and More</description> <pubDate></pubDate> <item> <title> fluent-package v5.0.5 has been released</title> <link>https://www.fluentd.org/blog/fluent-package-v5.0.5-has-been-released</link> <description><p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.5">v5.0.5</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.6 and Fluentd was updated to 1.16.6. We recommend upgrading to fluent-package v5.0.5!</p> <h3>Changes from fluent-package v5.0.4</h3> <p>In this release, focused on updating bundled ones.</p> <ul> <li>Update ruby to 3.2.6</li> <li>Update fluentd to 1.16.6</li> <li>Update bundled gems overall</li> <li>Solved possibility of DoS through &quot;NoMemoryError&quot;</li> <li>msi: set <code>GEM_HOME</code>/<code>GEM_PATH</code> in fluentd.bat</li> <li>Fixed SIGABORT error with fluent-plugin-systemd</li> </ul> <h3>Solved possibility of DoS through &quot;NoMemoryError&quot;</h3> <p>In the previous versions of msgpack gem, there was a possibility that cause DoS (Denial Of Service) when crafted message was sent to Fluentd.</p> <p>When that message was sent, it might cause &quot;NoMemoryError&quot; on Fluentd because there is a case that msgpack try to pre-allocate huge amount of memories.</p> <p>Above behavior was fixed not to pre-allocate over 32k entries (limit pre-allocated amount of memories) in msgpack 1.7.3 and bundled into fluent-package v5.0.5.</p> <h3>msi: set <code>GEM_HOME</code>/<code>GEM_PATH</code> in fluentd.bat</h3> <p>In the previous versions, fluentd.bat doesn&#39;t set GEM<em>HOME/GEM</em>PATH explicitly.</p> <p>If users set custom <code>GEM_HOME</code> / <code>GEM_PATH</code> environment variables, the batch file will not work as expected. Therefore, this patch will set the appropriate values in the batch file.</p> <p>Above bug was also fixed in fluent-package v5.0.5.</p> <h3>Fixed SIGABORT error with fluent-plugin-systemd</h3> <p>In this release, the bundled fluent-plugin-systemd was updated to 1.1.0. That release fixed SIGABORT error with inconsistency of memory allocator handling.</p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> <h4>About next LTS schedule</h4> <p>We plan to ship the next LTS version of fluent-package v5.0.6 on March, 2025. The content of updates are still in T.B.D.</p> <h4>End of support for td-agent v4, let&#39;s migrate to fluent-package</h4> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> </description> <pubDate>2024-11-08</pubDate> <content:encoded><p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.5">v5.0.5</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.6 and Fluentd was updated to 1.16.6. We recommend upgrading to fluent-package v5.0.5!</p> <h3>Changes from fluent-package v5.0.4</h3> <p>In this release, focused on updating bundled ones.</p> <ul> <li>Update ruby to 3.2.6</li> <li>Update fluentd to 1.16.6</li> <li>Update bundled gems overall</li> <li>Solved possibility of DoS through &quot;NoMemoryError&quot;</li> <li>msi: set <code>GEM_HOME</code>/<code>GEM_PATH</code> in fluentd.bat</li> <li>Fixed SIGABORT error with fluent-plugin-systemd</li> </ul> <h3>Solved possibility of DoS through &quot;NoMemoryError&quot;</h3> <p>In the previous versions of msgpack gem, there was a possibility that cause DoS (Denial Of Service) when crafted message was sent to Fluentd.</p> <p>When that message was sent, it might cause &quot;NoMemoryError&quot; on Fluentd because there is a case that msgpack try to pre-allocate huge amount of memories.</p> <p>Above behavior was fixed not to pre-allocate over 32k entries (limit pre-allocated amount of memories) in msgpack 1.7.3 and bundled into fluent-package v5.0.5.</p> <h3>msi: set <code>GEM_HOME</code>/<code>GEM_PATH</code> in fluentd.bat</h3> <p>In the previous versions, fluentd.bat doesn&#39;t set GEM<em>HOME/GEM</em>PATH explicitly.</p> <p>If users set custom <code>GEM_HOME</code> / <code>GEM_PATH</code> environment variables, the batch file will not work as expected. Therefore, this patch will set the appropriate values in the batch file.</p> <p>Above bug was also fixed in fluent-package v5.0.5.</p> <h3>Fixed SIGABORT error with fluent-plugin-systemd</h3> <p>In this release, the bundled fluent-plugin-systemd was updated to 1.1.0. That release fixed SIGABORT error with inconsistency of memory allocator handling.</p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> <h4>About next LTS schedule</h4> <p>We plan to ship the next LTS version of fluent-package v5.0.6 on March, 2025. The content of updates are still in T.B.D.</p> <h4>End of support for td-agent v4, let&#39;s migrate to fluent-package</h4> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> </content:encoded> </item> <item> <title> Scheduled support lifecycle announcement about Fluent Package v6</title> <link>https://www.fluentd.org/blog/fluent-package-v6-scheduled-lifecycle</link> <description><p>Hi users!</p> <p>We had launched <code>fluent-package</code> v5 series last year, recently shipped <a href="/blog/fluent-package-v5.0.4-has-been-released">v5.0.4</a> in LTS release channel and <a href="http://localhost:9395/blog/fluent-package-v5.1.0-has-been-released">v5.1.0</a> in normal release channel.</p> <p>In this blog article, we explains the planned next major updates - v6.0.0.</p> <h3>When the next LTS (Long Term Support) version available?</h3> <p>In short, we will ship fluent-package v6 in Aug, 2025.</p> <p>We keeps two release channels as follows:</p> <ul> <li>Normal release</li> <li>LTS (Long Term Support)</li> </ul> <p>Here is the difference of these channels.</p> <ul> <li>Normal release (6.x series) <ul> <li>Recommended for early adopters (eager to try new features of Fluentd)</li> <li>Rapid release development (planned semi-annually releases, but it may vary.)</li> <li>Fluentd will be upgraded occasionally <ul> <li>Minor upgrade will be applied. e.g. 1.18 =&gt; 1.19 and so on.</li> <li>e.g. v6.0.1 (Fluentd v1.18.1), v6.0.2 (Fluentd v1.18.2), ... v6.1.0 (Fluentd v1.19.0)</li> </ul></li> <li>The only latest version will be supported (bug fix will not be backported into older version)</li> </ul></li> <li>LTS (Long Term Support, 6.0.x series) <ul> <li>Recommended for enterprise services</li> <li>Security and bug fix release only</li> <li>Fluentd will be upgraded only in the teeny version and will stick to 1.18.x series. (T.B.D.) <ul> <li>Thus, the version number will be 6.0.x. (6.1 series will not be shipped for LTS channel)</li> <li>e.g. v6.0.1 (Fluentd v1.18.1), v6.0.2 (Fluentd v1.18.2), v6.0.3 (Fluentd v1.18.3) ... v6.0.x (Fluentd v1.18.x) and so on.</li> </ul></li> <li>Next major version will be <code>fluent-package</code> v7.0.0, and will be shipped in 2027 (T.B.D.).</li> </ul></li> </ul> <p>As for fluent-package v6,</p> <ul> <li>fluent-package v5 LTS will be supported until Dec, 2025. <ul> <li>We expect users upgrade from v5 to v6 during the grace period.</li> </ul></li> <li>Until 6.1.0 is released, normal release channel and LTS channel are the same package.</li> <li>Upgrade from v4 or older version is not supported. Upgrade to v5 first, then upgrade it to v6.</li> </ul> <p>Current releases and future plans:</p> <p><img src="/images/blog/20240829_fluent-package-v6-scheduled-lifecycle.png" alt=""></p>  <table border=1 width="100%"> <tr> <th style='background-color: #eee'>Release Version</th> <th style='background-color: #eee'>Release Date</th> <th style='background-color: #eee'>End of life</th> </tr> <tr> <td>v5.0.x (LTS)</td> <td>Aug, 2023</td> <td>Dec, 2025</td> </tr> <tr> <td>v6.0.0 (LTS)</td> <td>Aug, 2025</td> <td>Dec, 2027 (T.B.D.)</td> </tr> <tr> <td>v7.0.0 (LTS)</td> <td>Aug, 2027 (T.B.D.)</td> <td>T.B.D.</td> </tr> </table> <p>Happy logging!</p> </description> <pubDate>2024-08-29</pubDate> <content:encoded><p>Hi users!</p> <p>We had launched <code>fluent-package</code> v5 series last year, recently shipped <a href="/blog/fluent-package-v5.0.4-has-been-released">v5.0.4</a> in LTS release channel and <a href="http://localhost:9395/blog/fluent-package-v5.1.0-has-been-released">v5.1.0</a> in normal release channel.</p> <p>In this blog article, we explains the planned next major updates - v6.0.0.</p> <h3>When the next LTS (Long Term Support) version available?</h3> <p>In short, we will ship fluent-package v6 in Aug, 2025.</p> <p>We keeps two release channels as follows:</p> <ul> <li>Normal release</li> <li>LTS (Long Term Support)</li> </ul> <p>Here is the difference of these channels.</p> <ul> <li>Normal release (6.x series) <ul> <li>Recommended for early adopters (eager to try new features of Fluentd)</li> <li>Rapid release development (planned semi-annually releases, but it may vary.)</li> <li>Fluentd will be upgraded occasionally <ul> <li>Minor upgrade will be applied. e.g. 1.18 =&gt; 1.19 and so on.</li> <li>e.g. v6.0.1 (Fluentd v1.18.1), v6.0.2 (Fluentd v1.18.2), ... v6.1.0 (Fluentd v1.19.0)</li> </ul></li> <li>The only latest version will be supported (bug fix will not be backported into older version)</li> </ul></li> <li>LTS (Long Term Support, 6.0.x series) <ul> <li>Recommended for enterprise services</li> <li>Security and bug fix release only</li> <li>Fluentd will be upgraded only in the teeny version and will stick to 1.18.x series. (T.B.D.) <ul> <li>Thus, the version number will be 6.0.x. (6.1 series will not be shipped for LTS channel)</li> <li>e.g. v6.0.1 (Fluentd v1.18.1), v6.0.2 (Fluentd v1.18.2), v6.0.3 (Fluentd v1.18.3) ... v6.0.x (Fluentd v1.18.x) and so on.</li> </ul></li> <li>Next major version will be <code>fluent-package</code> v7.0.0, and will be shipped in 2027 (T.B.D.).</li> </ul></li> </ul> <p>As for fluent-package v6,</p> <ul> <li>fluent-package v5 LTS will be supported until Dec, 2025. <ul> <li>We expect users upgrade from v5 to v6 during the grace period.</li> </ul></li> <li>Until 6.1.0 is released, normal release channel and LTS channel are the same package.</li> <li>Upgrade from v4 or older version is not supported. Upgrade to v5 first, then upgrade it to v6.</li> </ul> <p>Current releases and future plans:</p> <p><img src="/images/blog/20240829_fluent-package-v6-scheduled-lifecycle.png" alt=""></p>  <table border=1 width="100%"> <tr> <th style='background-color: #eee'>Release Version</th> <th style='background-color: #eee'>Release Date</th> <th style='background-color: #eee'>End of life</th> </tr> <tr> <td>v5.0.x (LTS)</td> <td>Aug, 2023</td> <td>Dec, 2025</td> </tr> <tr> <td>v6.0.0 (LTS)</td> <td>Aug, 2025</td> <td>Dec, 2027 (T.B.D.)</td> </tr> <tr> <td>v7.0.0 (LTS)</td> <td>Aug, 2027 (T.B.D.)</td> <td>T.B.D.</td> </tr> </table> <p>Happy logging!</p> </content:encoded> </item> <item> <title> Fluentd v1.16.6 has been released</title> <link>https://www.fluentd.org/blog/fluentd-v1.16.6-has-been-released</link> <description><p>Hi users!</p> <p>We have released v1.16.6 on 2024-08-20. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/v1.16/CHANGELOG.md#release-v1166---20240816">here</a>.</p> <p>This release is a maintenance release of v1.16 series. In this release, some bugs were fixed with backporting from v1.17.x.</p> <h2>Bug Fixes</h2> <h3><code>yaml_parser</code>: <code>$log_level</code> notation in YAML was supported</h3> <p>In this release, we added a support for <code>$log_level</code> in <a href="https://docs.fluentd.org/configuration/config-file-yaml">YAML config file syntax</a>.</p> <p>In the previous versions, if you used <code>log_level</code> in YAML configuration, it causes a following warning:</p> <pre><code>[warn]: #0 &#39;log_level&#39; is deprecated parameter name. use &#39;@log_level&#39; instead. </code></pre> <p>But <code>@log_level</code> is invalid in YAML format. In YAML format, it should be <code>$log_level</code> as same as similar parameter such as <code>$tag</code> and <code>$type</code>, but not supported before.</p> <p>Since Fluentd v1.16.6, <code>$log_level</code> has been supported! (In v1.17.x, it was supported since v1.17.1)</p> <h3>fluentd command: Fix <code>--plugin</code> (-p) option not to overwrite default value</h3> <p>In this release, we have fixed a bug that fluentd command: <code>--plugin</code> (-p) option could not overwrite default value.</p> <p>This bug was unexpectedly introduced since v1.16.0.</p> <p>Now it was reverted to the original behavior.</p> <h3>logger: Fix LoadError with console gem v1.25</h3> <p>In this release, we fixed a LoadError with console gem v1.25 or later.</p> <p>Since console gem v1.25 or later, it&#39;s internal change causes a LoadError in Fluentd. This error breaks existing plugin such as fluent-plugin-prometheus which provides monitoring REST API.</p> <ul> <li><a href="https://github.com/fluent/fluentd/issues/4576">500 response when scraping prometheus metrics in kubernetes with istio service mesh</a></li> </ul> <p>With supporting console gem v1.25 or later, this issue has solved!</p> <p>Enjoy logging!</p> </description> <pubDate>2024-08-20</pubDate> <content:encoded><p>Hi users!</p> <p>We have released v1.16.6 on 2024-08-20. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/v1.16/CHANGELOG.md#release-v1166---20240816">here</a>.</p> <p>This release is a maintenance release of v1.16 series. In this release, some bugs were fixed with backporting from v1.17.x.</p> <h2>Bug Fixes</h2> <h3><code>yaml_parser</code>: <code>$log_level</code> notation in YAML was supported</h3> <p>In this release, we added a support for <code>$log_level</code> in <a href="https://docs.fluentd.org/configuration/config-file-yaml">YAML config file syntax</a>.</p> <p>In the previous versions, if you used <code>log_level</code> in YAML configuration, it causes a following warning:</p> <pre><code>[warn]: #0 &#39;log_level&#39; is deprecated parameter name. use &#39;@log_level&#39; instead. </code></pre> <p>But <code>@log_level</code> is invalid in YAML format. In YAML format, it should be <code>$log_level</code> as same as similar parameter such as <code>$tag</code> and <code>$type</code>, but not supported before.</p> <p>Since Fluentd v1.16.6, <code>$log_level</code> has been supported! (In v1.17.x, it was supported since v1.17.1)</p> <h3>fluentd command: Fix <code>--plugin</code> (-p) option not to overwrite default value</h3> <p>In this release, we have fixed a bug that fluentd command: <code>--plugin</code> (-p) option could not overwrite default value.</p> <p>This bug was unexpectedly introduced since v1.16.0.</p> <p>Now it was reverted to the original behavior.</p> <h3>logger: Fix LoadError with console gem v1.25</h3> <p>In this release, we fixed a LoadError with console gem v1.25 or later.</p> <p>Since console gem v1.25 or later, it&#39;s internal change causes a LoadError in Fluentd. This error breaks existing plugin such as fluent-plugin-prometheus which provides monitoring REST API.</p> <ul> <li><a href="https://github.com/fluent/fluentd/issues/4576">500 response when scraping prometheus metrics in kubernetes with istio service mesh</a></li> </ul> <p>With supporting console gem v1.25 or later, this issue has solved!</p> <p>Enjoy logging!</p> </content:encoded> </item> <item> <title> Fluentd v1.17.1 has been released</title> <link>https://www.fluentd.org/blog/fluentd-v1.17.1-has-been-released</link> <description><p>Hi users!</p> <p>We have released v1.17.1 on 2024-08-19. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1171---20240819">here</a>.</p> <p>This release is a new release of v1.17 series. In this release, we added some new options for some plugins and fixed bugs of Parser.</p> <h2>Enhancement</h2> <h3><code>yaml_parser</code>: <code>$log_level</code> notation in YAML was supported</h3> <p>In this release, we added a support for <code>$log_level</code> in <a href="https://docs.fluentd.org/configuration/config-file-yaml">YAML config file syntax</a>.</p> <p>In the previous versions, if you used <code>log_level</code> in YAML configuration, it causes a following warning:</p> <pre><code>[warn]: #0 &#39;log_level&#39; is deprecated parameter name. use &#39;@log_level&#39; instead. </code></pre> <p>But <code>@log_level</code> is invalid in YAML format. In YAML format, it should be <code>$log_level</code> as same as similar parameter such as <code>$tag</code> and <code>$type</code>, but not supported before.</p> <p>Since Fluentd v1.17.1, <code>$log_level</code> has been supported now!</p> <h3><code>out_http</code>: <code>compress</code> option was supported</h3> <p>In this release, we added a new option <a href="https://docs.fluentd.org/output/http#compress">compress</a> for the <a href="https://docs.fluentd.org/output/http">out_http</a> plugin.</p> <p>You can compress HTTP request body by specifying <code>compress gzip</code>. To keep compatibility, the default configuration is set as <code>compress text</code> (no compression).</p> <p>This feature was contributed by @rockliffelewis. Thanks!</p> <h3><code>in_sample</code>: the default behavior was changed</h3> <p>In this release, we changed the default behavior of <a href="https://docs.fluentd.org/input/sample">in_sample</a> plugin and added a new option <a href="https://docs.fluentd.org/input/sample#reuse_record">reuse_record</a> for it.</p> <p>The default behavior has changed to copy sample data to avoid the impact of destructive changes by subsequent plugins.</p> <p>In the previous versions, when <code>in_sample</code> plugin is used with the subsequent filter parser plugin which uses <code>remove_key_name_field</code>, it raises the following error repeatedly.</p> <pre><code>#0 dump an error event: error_class=ArgumentError error=&quot;message does not exist&quot; </code></pre> <p>This error occurs when <code>key_name</code> and <code>remove_key_name_field</code> removes key from record with destructive change in filter parser. It affects generated sample data after twice or later. (This error is not limited to filter parser plugin. The similar destructive change will be done with the subsequent plugins, this error is reproducible)</p> <p>The new default behavior (<code>reuse_record false</code>) avoids this problem, but it increases the load when generating large amounts of sample data. You can use this new parameter to have the same performance as before with <code>reuse_record</code>.</p> <h2>Bug Fixes</h2> <h3>logger: Fix LoadError with console gem v1.25</h3> <p>In this release, we fixed a LoadError with console gem v1.25 or later.</p> <p>Since console gem v1.25 or later, it&#39;s internal change causes a LoadError in Fluentd. This error breaks existing plugin such as fluent-plugin-prometheus which provides monitoring REST API.</p> <ul> <li><a href="https://github.com/fluent/fluentd/issues/4576">500 response when scraping prometheus metrics in kubernetes with istio service mesh</a></li> </ul> <p>With supporting console gem v1.25 or later, this issue has solved!</p> <p>Enjoy logging!</p> </description> <pubDate>2024-08-19</pubDate> <content:encoded><p>Hi users!</p> <p>We have released v1.17.1 on 2024-08-19. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1171---20240819">here</a>.</p> <p>This release is a new release of v1.17 series. In this release, we added some new options for some plugins and fixed bugs of Parser.</p> <h2>Enhancement</h2> <h3><code>yaml_parser</code>: <code>$log_level</code> notation in YAML was supported</h3> <p>In this release, we added a support for <code>$log_level</code> in <a href="https://docs.fluentd.org/configuration/config-file-yaml">YAML config file syntax</a>.</p> <p>In the previous versions, if you used <code>log_level</code> in YAML configuration, it causes a following warning:</p> <pre><code>[warn]: #0 &#39;log_level&#39; is deprecated parameter name. use &#39;@log_level&#39; instead. </code></pre> <p>But <code>@log_level</code> is invalid in YAML format. In YAML format, it should be <code>$log_level</code> as same as similar parameter such as <code>$tag</code> and <code>$type</code>, but not supported before.</p> <p>Since Fluentd v1.17.1, <code>$log_level</code> has been supported now!</p> <h3><code>out_http</code>: <code>compress</code> option was supported</h3> <p>In this release, we added a new option <a href="https://docs.fluentd.org/output/http#compress">compress</a> for the <a href="https://docs.fluentd.org/output/http">out_http</a> plugin.</p> <p>You can compress HTTP request body by specifying <code>compress gzip</code>. To keep compatibility, the default configuration is set as <code>compress text</code> (no compression).</p> <p>This feature was contributed by @rockliffelewis. Thanks!</p> <h3><code>in_sample</code>: the default behavior was changed</h3> <p>In this release, we changed the default behavior of <a href="https://docs.fluentd.org/input/sample">in_sample</a> plugin and added a new option <a href="https://docs.fluentd.org/input/sample#reuse_record">reuse_record</a> for it.</p> <p>The default behavior has changed to copy sample data to avoid the impact of destructive changes by subsequent plugins.</p> <p>In the previous versions, when <code>in_sample</code> plugin is used with the subsequent filter parser plugin which uses <code>remove_key_name_field</code>, it raises the following error repeatedly.</p> <pre><code>#0 dump an error event: error_class=ArgumentError error=&quot;message does not exist&quot; </code></pre> <p>This error occurs when <code>key_name</code> and <code>remove_key_name_field</code> removes key from record with destructive change in filter parser. It affects generated sample data after twice or later. (This error is not limited to filter parser plugin. The similar destructive change will be done with the subsequent plugins, this error is reproducible)</p> <p>The new default behavior (<code>reuse_record false</code>) avoids this problem, but it increases the load when generating large amounts of sample data. You can use this new parameter to have the same performance as before with <code>reuse_record</code>.</p> <h2>Bug Fixes</h2> <h3>logger: Fix LoadError with console gem v1.25</h3> <p>In this release, we fixed a LoadError with console gem v1.25 or later.</p> <p>Since console gem v1.25 or later, it&#39;s internal change causes a LoadError in Fluentd. This error breaks existing plugin such as fluent-plugin-prometheus which provides monitoring REST API.</p> <ul> <li><a href="https://github.com/fluent/fluentd/issues/4576">500 response when scraping prometheus metrics in kubernetes with istio service mesh</a></li> </ul> <p>With supporting console gem v1.25 or later, this issue has solved!</p> <p>Enjoy logging!</p> </content:encoded> </item> <item> <title> fluent-package v5.1.0 has been released</title> <link>https://www.fluentd.org/blog/fluent-package-v5.1.0-has-been-released</link> <description><p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.1.0">v5.1.0</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.x series. Bundled ruby version was upgraded to 3.2.5 and fluentd v1.17.0 was shipped!</p> <h3>Changes from fluent-package v5.0.4</h3> <p>In this release, focused on updating bundled ones.</p> <ul> <li>Update fluentd to 1.17.0</li> <li>Update ruby to 3.2.5</li> <li>Update bundled gems overall</li> </ul> <h3>Update bundled components overall</h3> <p>In LTS version, conservative updating policy is enabled, so updating gem is very limited to bug fix or security fix. In contrast to LTS version, v5.1.0 bundles more recent components because of standard version.</p> <p>As we already announced in <a href="/blog/fluent-package-scheduled-lifecycle">Scheduled support lifecycle announcement about Fluent Package</a> blog article, in normal release channel, we will ship the latest version (v1.17.0) of Fluentd.</p> <p>So if you want to try using latest version of Fluentd, v5.1.0 is one for you. If you use fluent-package in enterprise services, keep using fluent-package 5.0.x (LTS).</p> <div markdown="span" class="alert alert-info" role="alert"> <i class="fa fa-info-circle"></i> <b>Note:</b>In the previous versions, there is no different in normal release channel and LTS release channel. But from this release, v5.1.x will be shipped in normal release channel. v5.0.x will be shipped in LTS channel. </div> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> <h4>About next LTS schedule</h4> <p>We plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.</p> <h4>End of support for td-agent v4, let&#39;s migrate to fluent-package</h4> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> </description> <pubDate>2024-08-02</pubDate> <content:encoded><p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.1.0">v5.1.0</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.x series. Bundled ruby version was upgraded to 3.2.5 and fluentd v1.17.0 was shipped!</p> <h3>Changes from fluent-package v5.0.4</h3> <p>In this release, focused on updating bundled ones.</p> <ul> <li>Update fluentd to 1.17.0</li> <li>Update ruby to 3.2.5</li> <li>Update bundled gems overall</li> </ul> <h3>Update bundled components overall</h3> <p>In LTS version, conservative updating policy is enabled, so updating gem is very limited to bug fix or security fix. In contrast to LTS version, v5.1.0 bundles more recent components because of standard version.</p> <p>As we already announced in <a href="/blog/fluent-package-scheduled-lifecycle">Scheduled support lifecycle announcement about Fluent Package</a> blog article, in normal release channel, we will ship the latest version (v1.17.0) of Fluentd.</p> <p>So if you want to try using latest version of Fluentd, v5.1.0 is one for you. If you use fluent-package in enterprise services, keep using fluent-package 5.0.x (LTS).</p> <div markdown="span" class="alert alert-info" role="alert"> <i class="fa fa-info-circle"></i> <b>Note:</b>In the previous versions, there is no different in normal release channel and LTS release channel. But from this release, v5.1.x will be shipped in normal release channel. v5.0.x will be shipped in LTS channel. </div> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> <h4>About next LTS schedule</h4> <p>We plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.</p> <h4>End of support for td-agent v4, let&#39;s migrate to fluent-package</h4> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> </content:encoded> </item> <item> <title> fluent-package v5.0.4 has been released</title> <link>https://www.fluentd.org/blog/fluent-package-v5.0.4-has-been-released</link> <description><p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.4">v5.0.4</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.4 and a foolproof mechanism was implemented to prevent launching duplicated Fluentd instances, we recommend upgrading to fluent-package v5.0.4!</p> <h3>Changes from fluent-package v5.0.3</h3> <ul> <li>Update ruby to 3.2.4 (<a href="https://github.com/fluent/fluent-package-builder/pull/645">#645</a>)</li> <li>Fixed to prevent launching Fluentd wrongly if the service is already running (<a href="https://github.com/fluent/fluent-package-builder/pull/648">#648</a>,<a href="https://github.com/fluent/fluent-package-builder/pull/649">#649</a>)</li> <li>Added support for Ubuntu 24.04 (Noble Numbat)</li> </ul> <div markdown="span" class="alert alert-info" role="alert"> <i class="fa fa-info-circle"></i> <b>Note:</b> In this release, dropped CentOS 7 package because of EOL. </div> <h3>Fixed to prevent launching Fluentd wrongly if the service is already running</h3> <p>In this release, a foolproof mechanism was implemented to prevent launching Fluentd wrongly if the service is already running.</p> <p>As you know, you can check the version of Fluentd with <code>fluentd --version</code>, but there is a case that <code>fluentd -v</code> is executed wrongly to do it.</p> <p>When already running Fluentd as a service, <code>fluentd -v</code> launches a duplicated Fluentd instance with the same fluentd configuration. If you launch duplicated Fluentd instances, it causes the corruption of processing Fluentd buffer. To prevent such a situation, a foolproof was implemented now.</p> <p>For example, if Fluentd is running as a service, launching Fluentd causes an error to block it.</p> <p>Here is the example on Windows:</p> <pre><code>&gt; fluentd Error: Can&#39;t start duplicate Fluentd instance with the default config. To start Fluentd, please do one of the following: (Caution: Please be careful not to start multiple instances with the same config.) - Stop the Fluentd Windows service &#39;fluentdwinsvc&#39;. - Specify the config path explicitly by &#39;-c&#39; (&#39;--config&#39;). </code></pre> <p>Even though if you wrongly launch Fluentd to check version with <code>-v</code> (It should be <code>--version</code> to show version), then it causes the following error.</p> <pre><code>&gt; fluentd -v Error: Can&#39;t start duplicate Fluentd instance with the default config. To take the version, please use &#39;--version&#39;, not &#39;-v&#39; (&#39;--verbose&#39;). To start Fluentd, please do one of the following: (Caution: Please be careful not to start multiple instances with the same config.) - Stop the Fluentd Windows service &#39;fluentdwinsvc&#39;. - Specify the config path explicitly by &#39;-c&#39; (&#39;--config&#39;) </code></pre> <p>Note that this foolproof feature is intended to block launching duplicated Fluentd instance, you can explicitly launch Fluentd by specifying a specific option to pass it even though already Fluentd is running as a service.</p> <p>On Windows:</p> <ul> <li><code>-c</code> (<code>--config</code>)</li> <li><code>--dry-run</code></li> <li><code>--reg-winsvc</code></li> <li><code>--reg-winsvc-fluentdopt</code></li> <li><code>--show-plugin-config</code></li> </ul> <p>On Linux:</p> <ul> <li><code>-c</code> (<code>--config</code>)</li> <li><code>--dry-run</code></li> <li><code>--show-plugin-config</code></li> </ul> <h3>About next LTS schedule</h3> <p>We plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.</p> <h3>End of support for td-agent v4, let&#39;s migrate to fluent-package</h3> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> </description> <pubDate>2024-07-02</pubDate> <content:encoded><p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.4">v5.0.4</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.4 and a foolproof mechanism was implemented to prevent launching duplicated Fluentd instances, we recommend upgrading to fluent-package v5.0.4!</p> <h3>Changes from fluent-package v5.0.3</h3> <ul> <li>Update ruby to 3.2.4 (<a href="https://github.com/fluent/fluent-package-builder/pull/645">#645</a>)</li> <li>Fixed to prevent launching Fluentd wrongly if the service is already running (<a href="https://github.com/fluent/fluent-package-builder/pull/648">#648</a>,<a href="https://github.com/fluent/fluent-package-builder/pull/649">#649</a>)</li> <li>Added support for Ubuntu 24.04 (Noble Numbat)</li> </ul> <div markdown="span" class="alert alert-info" role="alert"> <i class="fa fa-info-circle"></i> <b>Note:</b> In this release, dropped CentOS 7 package because of EOL. </div> <h3>Fixed to prevent launching Fluentd wrongly if the service is already running</h3> <p>In this release, a foolproof mechanism was implemented to prevent launching Fluentd wrongly if the service is already running.</p> <p>As you know, you can check the version of Fluentd with <code>fluentd --version</code>, but there is a case that <code>fluentd -v</code> is executed wrongly to do it.</p> <p>When already running Fluentd as a service, <code>fluentd -v</code> launches a duplicated Fluentd instance with the same fluentd configuration. If you launch duplicated Fluentd instances, it causes the corruption of processing Fluentd buffer. To prevent such a situation, a foolproof was implemented now.</p> <p>For example, if Fluentd is running as a service, launching Fluentd causes an error to block it.</p> <p>Here is the example on Windows:</p> <pre><code>&gt; fluentd Error: Can&#39;t start duplicate Fluentd instance with the default config. To start Fluentd, please do one of the following: (Caution: Please be careful not to start multiple instances with the same config.) - Stop the Fluentd Windows service &#39;fluentdwinsvc&#39;. - Specify the config path explicitly by &#39;-c&#39; (&#39;--config&#39;). </code></pre> <p>Even though if you wrongly launch Fluentd to check version with <code>-v</code> (It should be <code>--version</code> to show version), then it causes the following error.</p> <pre><code>&gt; fluentd -v Error: Can&#39;t start duplicate Fluentd instance with the default config. To take the version, please use &#39;--version&#39;, not &#39;-v&#39; (&#39;--verbose&#39;). To start Fluentd, please do one of the following: (Caution: Please be careful not to start multiple instances with the same config.) - Stop the Fluentd Windows service &#39;fluentdwinsvc&#39;. - Specify the config path explicitly by &#39;-c&#39; (&#39;--config&#39;) </code></pre> <p>Note that this foolproof feature is intended to block launching duplicated Fluentd instance, you can explicitly launch Fluentd by specifying a specific option to pass it even though already Fluentd is running as a service.</p> <p>On Windows:</p> <ul> <li><code>-c</code> (<code>--config</code>)</li> <li><code>--dry-run</code></li> <li><code>--reg-winsvc</code></li> <li><code>--reg-winsvc-fluentdopt</code></li> <li><code>--show-plugin-config</code></li> </ul> <p>On Linux:</p> <ul> <li><code>-c</code> (<code>--config</code>)</li> <li><code>--dry-run</code></li> <li><code>--show-plugin-config</code></li> </ul> <h3>About next LTS schedule</h3> <p>We plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.</p> <h3>End of support for td-agent v4, let&#39;s migrate to fluent-package</h3> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> </content:encoded> </item> <item> <title> Fluentd v1.17.0 has been released</title> <link>https://www.fluentd.org/blog/fluentd-v1.17.0-has-been-released</link> <description><p>Hi users!</p> <p>We have released v1.17.0 on 2024-04-30. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430">here</a>.</p> <p>This release is a new release of v1.17 series. In this release, we added some new features for some plugins and fixed bugs of Parser.</p> <h2>Enhancement</h2> <h3><code>in_tail</code>: Add <code>glob_policy</code> option for expanding glob capability of <code>path</code> and <code>exclude_path</code></h3> <p>In this release, we added a new option <a href="https://docs.fluentd.org/input/tail#glob_policy">glob_policy</a> for <a href="https://docs.fluentd.org/input/tail">in_tail</a> plugin.</p> <p>In previous versions, we can use only <code>*</code> in glob patterns for <a href="https://docs.fluentd.org/input/tail#path">path</a> and <a href="https://docs.fluentd.org/input/tail#exclude_path">exclude_path</a> option.</p> <p>Example:</p> <pre><code>path /path/to/* exclude_path [&quot;/path/to/*.gz&quot;, &quot;/path/to/*.zip&quot;] </code></pre> <p>From this version, we can also use <code>[]</code>, <code>?</code>, and <code>{}</code> in glob patterns depending on the <code>glob_policy</code> option.</p> <p>Example:</p> <pre><code>path &quot;[0-1][2-3].log&quot; glob_policy extended </code></pre> <p>Please see <a href="https://docs.fluentd.org/input/tail#glob_policy">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4401">#4401</a> for more information.</p> <h3><code>out_http</code>: Support AWS Signature Version 4 authentication</h3> <p>In this release, we added a new option <code>aws_sigv4</code> for the <a href="https://docs.fluentd.org/output/http#method">method</a> setting of <a href="https://docs.fluentd.org/output/http">out_http</a> plugin.</p> <p>By using this option, <code>out_http</code> can use <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html">AWS Signature Version 4</a>.</p> <p>For example, this allows <code>out_http</code> to write to <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html">Amazon OpenSearch Ingestion</a>.</p> <p>Please see <a href="https://docs.fluentd.org/output/http#method">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4459">#4459</a> for more information.</p> <h3><code>out_http</code>: Add option to reuse connections</h3> <p>In this release, we add a new option <a href="https://docs.fluentd.org/output/http#reuse_connections">reuse_connections</a> for <a href="https://docs.fluentd.org/output/http">out_http</a> plugin.</p> <p>This option will improve throughput of <code>out_http</code>.</p> <p>Please see <a href="https://docs.fluentd.org/output/http#reuse_connections">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4330">#4330</a> for more information.</p> <h3><code>in_http</code>: Recognize CSP reports as JSON data</h3> <p>In this release, we make the data type of the request where the <code>Content-Type</code> is <code>application/csp-report</code> be considered JSON by default.</p> <p>Now, <code>in_http</code> can receive <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP">Content Security Policy</a>&#39;s report by default.</p> <p>Please see <a href="https://github.com/fluent/fluentd/pull/4282">#4282</a> for more information.</p> <h2>Bug Fixes</h2> <h3>Make sure parser returns hash</h3> <p>The record data in an event of Fluentd must be a hash object.</p> <ul> <li><a href="https://docs.fluentd.org/quickstart/life-of-a-fluentd-event">Life of a Fluentd event</a></li> </ul> <p>However, in the previous versions, some parser plugins could return a non-hash object, such as an array. It could cause errors in subsequent processing.</p> <p>In this release, the following parser plugins have been fixed.</p> <ul> <li><a href="https://docs.fluentd.org/parser/json">parser_json</a></li> <li><a href="https://docs.fluentd.org/parser/msgpack">parser_msgpack</a></li> </ul> <p>The changes are as follows:</p> <ul> <li>Make sure to return a hash record.</li> <li>Make sure to accept only a hash or an array of hash.</li> </ul> <p>Here are the details for each case.</p> <h4>Example of changed behavior</h4> <p>Config:</p> <pre><code>&lt;source&gt; @type tcp tag test.tcp &lt;parse&gt; @type json &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>Send an array data:</p> <pre><code>$ netcat 0.0.0.0 5170 [{&quot;k&quot;:&quot;v&quot;}, {&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result before this version:</p> <pre><code>{datetime} test.tcp: [{&quot;k&quot;:&quot;v&quot;},{&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result after this version:</p> <pre><code>{datetime} test.tcp: {&quot;k&quot;:&quot;v&quot;} {datetime} test.tcp: {&quot;k2&quot;:&quot;v2&quot;} </code></pre> <h4>Example of resolved error</h4> <p>Config:</p> <pre><code>&lt;source&gt; @type tcp tag test.tcp &lt;parse&gt; @type json null_empty_string &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>Send an array data:</p> <pre><code>$ netcat 0.0.0.0 5170 [{&quot;k&quot;:&quot;v&quot;}, {&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result before this version:</p> <pre><code>{datetime} [error]: #0 unexpected error on reading data host=&quot;xxx&quot; port=xxx error_class=NoMethodError error=&quot;undefined method `each_key&#39; for [{\&quot;k\&quot;=&gt;\&quot;v\&quot;}, {\&quot;k2\&quot;=&gt;\&quot;v2\&quot;}]:Array&quot; </code></pre> <p>The result after this version:</p> <pre><code>{datetime} test.tcp: {&quot;k&quot;:&quot;v&quot;} {datetime} test.tcp: {&quot;k2&quot;:&quot;v2&quot;} </code></pre> <h4>Remaining problem: filter_parser</h4> <p>In the previous versions, <code>filter_parser</code> could return an array record based on this wrong behavior. From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case. This behavior should improve in the future.</p> <p>Here is an example.</p> <pre><code>&lt;source&gt; @type sample tag test.array sample {&quot;message&quot;: &quot;[{\&quot;k\&quot;:\&quot;v\&quot;}, {\&quot;k2\&quot;:\&quot;v2\&quot;}]&quot;} &lt;/source&gt; &lt;filter test.**&gt; @type parser key_name message &lt;parse&gt; @type json &lt;/parse&gt; &lt;/filter&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>The result before this version:</p> <pre><code>{datetime} test.array: [{&quot;k&quot;:&quot;v&quot;},{&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result after this version:</p> <pre><code>{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=&quot;Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: &#39;[{\&quot;k\&quot;:\&quot;v\&quot;}, {\&quot;k2\&quot;:\&quot;v2\&quot;}]&#39;&quot; location=nil tag=&quot;test.array&quot; time=xxx record={&quot;k2&quot;=&gt;&quot;v2&quot;} {datetime} test.array: {&quot;k&quot;:&quot;v&quot;} </code></pre> <p>These are the major changes for this release.</p> <p>In addition, some performance improvements have been included! Please see <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430">ChangeLog</a> for details!</p> <p>Enjoy logging!</p> </description> <pubDate>2024-04-30</pubDate> <content:encoded><p>Hi users!</p> <p>We have released v1.17.0 on 2024-04-30. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430">here</a>.</p> <p>This release is a new release of v1.17 series. In this release, we added some new features for some plugins and fixed bugs of Parser.</p> <h2>Enhancement</h2> <h3><code>in_tail</code>: Add <code>glob_policy</code> option for expanding glob capability of <code>path</code> and <code>exclude_path</code></h3> <p>In this release, we added a new option <a href="https://docs.fluentd.org/input/tail#glob_policy">glob_policy</a> for <a href="https://docs.fluentd.org/input/tail">in_tail</a> plugin.</p> <p>In previous versions, we can use only <code>*</code> in glob patterns for <a href="https://docs.fluentd.org/input/tail#path">path</a> and <a href="https://docs.fluentd.org/input/tail#exclude_path">exclude_path</a> option.</p> <p>Example:</p> <pre><code>path /path/to/* exclude_path [&quot;/path/to/*.gz&quot;, &quot;/path/to/*.zip&quot;] </code></pre> <p>From this version, we can also use <code>[]</code>, <code>?</code>, and <code>{}</code> in glob patterns depending on the <code>glob_policy</code> option.</p> <p>Example:</p> <pre><code>path &quot;[0-1][2-3].log&quot; glob_policy extended </code></pre> <p>Please see <a href="https://docs.fluentd.org/input/tail#glob_policy">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4401">#4401</a> for more information.</p> <h3><code>out_http</code>: Support AWS Signature Version 4 authentication</h3> <p>In this release, we added a new option <code>aws_sigv4</code> for the <a href="https://docs.fluentd.org/output/http#method">method</a> setting of <a href="https://docs.fluentd.org/output/http">out_http</a> plugin.</p> <p>By using this option, <code>out_http</code> can use <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html">AWS Signature Version 4</a>.</p> <p>For example, this allows <code>out_http</code> to write to <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html">Amazon OpenSearch Ingestion</a>.</p> <p>Please see <a href="https://docs.fluentd.org/output/http#method">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4459">#4459</a> for more information.</p> <h3><code>out_http</code>: Add option to reuse connections</h3> <p>In this release, we add a new option <a href="https://docs.fluentd.org/output/http#reuse_connections">reuse_connections</a> for <a href="https://docs.fluentd.org/output/http">out_http</a> plugin.</p> <p>This option will improve throughput of <code>out_http</code>.</p> <p>Please see <a href="https://docs.fluentd.org/output/http#reuse_connections">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4330">#4330</a> for more information.</p> <h3><code>in_http</code>: Recognize CSP reports as JSON data</h3> <p>In this release, we make the data type of the request where the <code>Content-Type</code> is <code>application/csp-report</code> be considered JSON by default.</p> <p>Now, <code>in_http</code> can receive <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP">Content Security Policy</a>&#39;s report by default.</p> <p>Please see <a href="https://github.com/fluent/fluentd/pull/4282">#4282</a> for more information.</p> <h2>Bug Fixes</h2> <h3>Make sure parser returns hash</h3> <p>The record data in an event of Fluentd must be a hash object.</p> <ul> <li><a href="https://docs.fluentd.org/quickstart/life-of-a-fluentd-event">Life of a Fluentd event</a></li> </ul> <p>However, in the previous versions, some parser plugins could return a non-hash object, such as an array. It could cause errors in subsequent processing.</p> <p>In this release, the following parser plugins have been fixed.</p> <ul> <li><a href="https://docs.fluentd.org/parser/json">parser_json</a></li> <li><a href="https://docs.fluentd.org/parser/msgpack">parser_msgpack</a></li> </ul> <p>The changes are as follows:</p> <ul> <li>Make sure to return a hash record.</li> <li>Make sure to accept only a hash or an array of hash.</li> </ul> <p>Here are the details for each case.</p> <h4>Example of changed behavior</h4> <p>Config:</p> <pre><code>&lt;source&gt; @type tcp tag test.tcp &lt;parse&gt; @type json &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>Send an array data:</p> <pre><code>$ netcat 0.0.0.0 5170 [{&quot;k&quot;:&quot;v&quot;}, {&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result before this version:</p> <pre><code>{datetime} test.tcp: [{&quot;k&quot;:&quot;v&quot;},{&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result after this version:</p> <pre><code>{datetime} test.tcp: {&quot;k&quot;:&quot;v&quot;} {datetime} test.tcp: {&quot;k2&quot;:&quot;v2&quot;} </code></pre> <h4>Example of resolved error</h4> <p>Config:</p> <pre><code>&lt;source&gt; @type tcp tag test.tcp &lt;parse&gt; @type json null_empty_string &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>Send an array data:</p> <pre><code>$ netcat 0.0.0.0 5170 [{&quot;k&quot;:&quot;v&quot;}, {&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result before this version:</p> <pre><code>{datetime} [error]: #0 unexpected error on reading data host=&quot;xxx&quot; port=xxx error_class=NoMethodError error=&quot;undefined method `each_key&#39; for [{\&quot;k\&quot;=&gt;\&quot;v\&quot;}, {\&quot;k2\&quot;=&gt;\&quot;v2\&quot;}]:Array&quot; </code></pre> <p>The result after this version:</p> <pre><code>{datetime} test.tcp: {&quot;k&quot;:&quot;v&quot;} {datetime} test.tcp: {&quot;k2&quot;:&quot;v2&quot;} </code></pre> <h4>Remaining problem: filter_parser</h4> <p>In the previous versions, <code>filter_parser</code> could return an array record based on this wrong behavior. From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case. This behavior should improve in the future.</p> <p>Here is an example.</p> <pre><code>&lt;source&gt; @type sample tag test.array sample {&quot;message&quot;: &quot;[{\&quot;k\&quot;:\&quot;v\&quot;}, {\&quot;k2\&quot;:\&quot;v2\&quot;}]&quot;} &lt;/source&gt; &lt;filter test.**&gt; @type parser key_name message &lt;parse&gt; @type json &lt;/parse&gt; &lt;/filter&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>The result before this version:</p> <pre><code>{datetime} test.array: [{&quot;k&quot;:&quot;v&quot;},{&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result after this version:</p> <pre><code>{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=&quot;Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: &#39;[{\&quot;k\&quot;:\&quot;v\&quot;}, {\&quot;k2\&quot;:\&quot;v2\&quot;}]&#39;&quot; location=nil tag=&quot;test.array&quot; time=xxx record={&quot;k2&quot;=&gt;&quot;v2&quot;} {datetime} test.array: {&quot;k&quot;:&quot;v&quot;} </code></pre> <p>These are the major changes for this release.</p> <p>In addition, some performance improvements have been included! Please see <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430">ChangeLog</a> for details!</p> <p>Enjoy logging!</p> </content:encoded> </item> <item> <title> fluent-package v5.0.3 has been released</title> <link>https://www.fluentd.org/blog/fluent-package-v5.0.3-has-been-released</link> <description><p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.3">v5.0.3</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.0.x LTS series. As significant slow starting service and crash issues during startup on Windows were fixed, we recommend upgrading to fluent-package v5.0.3!</p> <h3>Changes from fluent-package v5.0.2</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.16.5">1.16.5</a>. See the following blog articles about details. <ul> <li><a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a></li> <li><a href="fluentd-v1.16.5-have-been-released">Fluentd v1.16.5 has been released</a></li> </ul></li> <li>Update bundled plugins <ul> <li>e.g. fluent-diagtool v1.0.5. It supports to collect list of plugins on Windows.</li> </ul></li> <li>msi: fixed wrong environment path for Fluent Package Prompt (<a href="https://github.com/fluent/fluent-package-builder/pull/606">#606</a>) <ul> <li>It breaks fluent-diagtool behavior to launch fluent-gem correctly.</li> </ul></li> <li>msi: removed unnecessary path delimiter (<a href="https://github.com/fluent/fluent-package-builder/pull/607">#607</a>) <ul> <li>It doesn&#39;t cause any problem yet, but it should treat <code>%~dp0</code> correctly.</li> </ul></li> <li>rpm: fixed to take over enabled state of systemd service from td-agent v4 (<a href="https://github.com/fluent/fluent-package-builder/pull/613">#613</a>)</li> <li>deb rpm: fixed to quote target files correctly not to cause migration failures (<a href="https://github.com/fluent/fluent-package-builder/pull/615">#615</a>)</li> <li>msi: added a patch for RubyInstaller to avoid crash on start up (<a href="https://github.com/fluent/fluent-package-builder/pull/620">#620</a>)</li> <li>msi: fixed slow start issue on Windows (<a href="https://github.com/fluent/fluent-package-builder/pull/631">#631</a>)</li> </ul> <div markdown="span" class="alert alert-info" role="alert"> <i class="fa fa-info-circle"></i> <b>Note:</b> v5.0.3 for Windows msi will be shipped later. </div> <h3>About next LTS schedule</h3> <p>We plan to ship next LTS version of fluent-package v5.0.4 on June, 2024. The content of updates are still in T.B.D.</p> <h3>End of support for td-agent v4, let&#39;s migrate to fluent-package</h3> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> </description> <pubDate>2024-03-29</pubDate> <content:encoded><p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.3">v5.0.3</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.0.x LTS series. As significant slow starting service and crash issues during startup on Windows were fixed, we recommend upgrading to fluent-package v5.0.3!</p> <h3>Changes from fluent-package v5.0.2</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.16.5">1.16.5</a>. See the following blog articles about details. <ul> <li><a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a></li> <li><a href="fluentd-v1.16.5-have-been-released">Fluentd v1.16.5 has been released</a></li> </ul></li> <li>Update bundled plugins <ul> <li>e.g. fluent-diagtool v1.0.5. It supports to collect list of plugins on Windows.</li> </ul></li> <li>msi: fixed wrong environment path for Fluent Package Prompt (<a href="https://github.com/fluent/fluent-package-builder/pull/606">#606</a>) <ul> <li>It breaks fluent-diagtool behavior to launch fluent-gem correctly.</li> </ul></li> <li>msi: removed unnecessary path delimiter (<a href="https://github.com/fluent/fluent-package-builder/pull/607">#607</a>) <ul> <li>It doesn&#39;t cause any problem yet, but it should treat <code>%~dp0</code> correctly.</li> </ul></li> <li>rpm: fixed to take over enabled state of systemd service from td-agent v4 (<a href="https://github.com/fluent/fluent-package-builder/pull/613">#613</a>)</li> <li>deb rpm: fixed to quote target files correctly not to cause migration failures (<a href="https://github.com/fluent/fluent-package-builder/pull/615">#615</a>)</li> <li>msi: added a patch for RubyInstaller to avoid crash on start up (<a href="https://github.com/fluent/fluent-package-builder/pull/620">#620</a>)</li> <li>msi: fixed slow start issue on Windows (<a href="https://github.com/fluent/fluent-package-builder/pull/631">#631</a>)</li> </ul> <div markdown="span" class="alert alert-info" role="alert"> <i class="fa fa-info-circle"></i> <b>Note:</b> v5.0.3 for Windows msi will be shipped later. </div> <h3>About next LTS schedule</h3> <p>We plan to ship next LTS version of fluent-package v5.0.4 on June, 2024. The content of updates are still in T.B.D.</p> <h3>End of support for td-agent v4, let&#39;s migrate to fluent-package</h3> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> </content:encoded> </item> <item> <title> Fluentd v1.16.5 has been released</title> <link>https://www.fluentd.org/blog/fluentd-v1.16.5-have-been-released</link> <description><p>Hi users!</p> <p>We have released v1.16.5 on 2024-03-27. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1165---20240327">here</a>.</p> <p>The previous version v1.16.4 has a serious Buffer bug when it processes large data exceeding chunk size limit. So, we have released v1.16.5 urgently, and have fixed the bug. So, <strong>please don&#39;t use v1.16.4</strong>.</p> <p>Sorry for troubling.</p> <h2>Bug Fixes</h2> <h3>Buffer: Fix emit error of v1.16.4 sometimes failing to process large data exceeding chunk size limit</h3> <p>In the previous version v1.16.4, we fixed a Buffer bug (Please see <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a> for details).</p> <p>There was a race condition problem with that fix, and similar Buffer errors explained in <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a> could occur.</p> <ul> <li><code>emit transaction failed: ...</code></li> <li><code>send an error event stream to @ERROR: ...</code></li> </ul> <p>These errors are the same as the Buffer error of <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a>, but it would be more likely to happen. The cause is a race condition when processing large data exceeding chunk size limit. So, in that case, these errors can occur depending on the timing.</p> <p>So, please don&#39;t use v1.16.4.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/4447">#4447</a> for more information.</p> <p>Sorry for troubling.</p> </description> <pubDate>2024-03-27</pubDate> <content:encoded><p>Hi users!</p> <p>We have released v1.16.5 on 2024-03-27. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1165---20240327">here</a>.</p> <p>The previous version v1.16.4 has a serious Buffer bug when it processes large data exceeding chunk size limit. So, we have released v1.16.5 urgently, and have fixed the bug. So, <strong>please don&#39;t use v1.16.4</strong>.</p> <p>Sorry for troubling.</p> <h2>Bug Fixes</h2> <h3>Buffer: Fix emit error of v1.16.4 sometimes failing to process large data exceeding chunk size limit</h3> <p>In the previous version v1.16.4, we fixed a Buffer bug (Please see <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a> for details).</p> <p>There was a race condition problem with that fix, and similar Buffer errors explained in <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a> could occur.</p> <ul> <li><code>emit transaction failed: ...</code></li> <li><code>send an error event stream to @ERROR: ...</code></li> </ul> <p>These errors are the same as the Buffer error of <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a>, but it would be more likely to happen. The cause is a race condition when processing large data exceeding chunk size limit. So, in that case, these errors can occur depending on the timing.</p> <p>So, please don&#39;t use v1.16.4.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/4447">#4447</a> for more information.</p> <p>Sorry for troubling.</p> </content:encoded> </item> <item> <title> Fluentd v1.16.4 has been released</title> <link>https://www.fluentd.org/blog/fluentd-v1.16.4-have-been-released</link> <description><p>Hi users!</p> <p>We have released v1.16.4 on 2024-03-14. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1164---20240314">here</a>.</p> <p>In this release, we fixed several bugs and improved performance.</p> <h2>Bug Fixes</h2> <h3>Buffer: Fix emit error sometimes caused by failing to process large data exceeding chunk size limit</h3> <p>In previous versions, when Buffer processes large data exceeding the chunk size limit, it sometimes fails and causes an emit error.</p> <p>It is usually unlikely, but it may occur when Buffer receives data with extremely large records that are distributed unevenly in the data.</p> <p>If you set <a href="https://docs.fluentd.org/configuration/config-file#error-label">@ERROR</a> label, the data is routed to that label, and Fluentd outputs a warning log message: <code>send an error event stream to @ERROR: ...</code>.</p> <p>However, if you don&#39;t set <code>@ERROR</code> label, <strong>the data is discarded</strong>, and Fluentd outputs a warning log message: <code>emit transaction failed: ...</code>. In addition, Input plugins that do not consider an emit error <strong>may stop working</strong>, such as <code>in_windows_eventlog2</code>.</p> <p>Here is an example of the warning message:</p> <pre><code>2024-03-22 14:13:35 +0900 [warn]: #0 emit transaction failed: error_class=IOError error=&quot;closed stream&quot; location=&quot;/path/to/fluentd/lib/fluent/plugin/buffer/file_chunk.rb:82:in `pos&#39;&quot; tag=&quot;test&quot; </code></pre> <p>If you see these warnings, please update Fluentd.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/4342">#4342</a> for more information.</p> <h3>Others</h3> <ul> <li><code>in_tail</code>: Fix tail watchers in <code>rotate_wait</code> state not being managed. <a href="https://github.com/fluent/fluentd/pull/4334">#4334</a> <ul> <li>This problem should not have any actual negative impact on the operation.</li> </ul></li> <li>Buffer: Avoid unnecessary log processing. It will improve performance. <a href="https://github.com/fluent/fluentd/pull/4331">#4331</a></li> </ul> <p>Enjoy logging!</p> </description> <pubDate>2024-03-14</pubDate> <content:encoded><p>Hi users!</p> <p>We have released v1.16.4 on 2024-03-14. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1164---20240314">here</a>.</p> <p>In this release, we fixed several bugs and improved performance.</p> <h2>Bug Fixes</h2> <h3>Buffer: Fix emit error sometimes caused by failing to process large data exceeding chunk size limit</h3> <p>In previous versions, when Buffer processes large data exceeding the chunk size limit, it sometimes fails and causes an emit error.</p> <p>It is usually unlikely, but it may occur when Buffer receives data with extremely large records that are distributed unevenly in the data.</p> <p>If you set <a href="https://docs.fluentd.org/configuration/config-file#error-label">@ERROR</a> label, the data is routed to that label, and Fluentd outputs a warning log message: <code>send an error event stream to @ERROR: ...</code>.</p> <p>However, if you don&#39;t set <code>@ERROR</code> label, <strong>the data is discarded</strong>, and Fluentd outputs a warning log message: <code>emit transaction failed: ...</code>. In addition, Input plugins that do not consider an emit error <strong>may stop working</strong>, such as <code>in_windows_eventlog2</code>.</p> <p>Here is an example of the warning message:</p> <pre><code>2024-03-22 14:13:35 +0900 [warn]: #0 emit transaction failed: error_class=IOError error=&quot;closed stream&quot; location=&quot;/path/to/fluentd/lib/fluent/plugin/buffer/file_chunk.rb:82:in `pos&#39;&quot; tag=&quot;test&quot; </code></pre> <p>If you see these warnings, please update Fluentd.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/4342">#4342</a> for more information.</p> <h3>Others</h3> <ul> <li><code>in_tail</code>: Fix tail watchers in <code>rotate_wait</code> state not being managed. <a href="https://github.com/fluent/fluentd/pull/4334">#4334</a> <ul> <li>This problem should not have any actual negative impact on the operation.</li> </ul></li> <li>Buffer: Avoid unnecessary log processing. It will improve performance. <a href="https://github.com/fluent/fluentd/pull/4331">#4331</a></li> </ul> <p>Enjoy logging!</p> </content:encoded> </item> </channel> </rss>

CINXE.COM

Fluentd Blog