CINXE.COM

<!DOCTYPE html> <html lang="en-us"> <head> <meta charset="UTF-8"> <title>True streaming for libvips</title> <meta name="description" content="A fast image processing library with low memory needs."/> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="theme-color" content="#157878"> <link href='https://fonts.googleapis.com/css?family=Open+Sans:400,700' rel='stylesheet' type='text/css'> <link href='https://fonts.googleapis.com/css?family=Cabin:400,700' rel='stylesheet' type='text/css'> <link rel="stylesheet" href="/assets/css/style.css?v=e926315de91139ac2a6fc1a99556e1fe8f3ddce3"> <link rel="shortcut icon" type="image/x-icon" href="favicon.ico"> <link type="application/atom+xml" rel="alternate" href="https://www.libvips.org/feed.xml" title="libvips" /> </head> <body> <section class="page-header"> <h1 class="project-name"><a href="/">libvips</a></h1> <h2 class="project-tagline">A fast image processing library with low memory needs.</h2> <a href="https://github.com/libvips/libvips/releases" class="btn">Download</a> <a href="/install.html" class="btn">Install</a> <a href="/API/current" class="btn">Documentation</a> <a href="https://github.com/libvips/libvips/issues" class="btn">Issues</a> <a href="https://github.com/libvips/libvips/wiki" class="btn">Wiki</a> <a href="https://github.com/libvips" class="btn">libvips projects</a> <a href="https://github.com/libvips/libvips" class="btn">libvips on GitHub</a> </section> <section class="main-content"> An interesting feature has just landed in libvips git master (and should be in the upcoming libvips 8.9): true streaming. This has been talked about on and off for five years or more, but it鈥檚 now finally happened! This post explains what this feature is and why it could be useful. <h1 id="overview">Overview</h1> Previously, libvips let you use files and areas of memory as the source and destination of image processing pipelines. The new <code class="language-plaintext highlighter-rouge">VipsConnection</code> classes let you connect image processing pipelines efficiently to any kind of data object, for example, pipes. You can now do this: <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat k2.jpg | \ vips invert stdin[shrink=2] .jpg[Q=90] | \ cat > x.jpg </code></pre></div></div> The magic filename <code class="language-plaintext highlighter-rouge">"stdin"</code> opens a stream attached to file descriptor 0 (<code class="language-plaintext highlighter-rouge">stdin</code>), does <code class="language-plaintext highlighter-rouge">vips_image_new_from_source()</code>, and passes that image into the operation. Writing to a filename with nothing before the suffix will open a stream to <code class="language-plaintext highlighter-rouge">stdout</code> and write in that format. <h1 id="why-is-this-useful">Why is this useful</h1> To see why this is a useful thing to be able to do, imagine how something like a thumbnailing service on S3 works. S3 keeps data (images in this case) in buckets and lets you read and write buckets using http <code class="language-plaintext highlighter-rouge">GET</code> and <code class="language-plaintext highlighter-rouge">POST</code> requests to addresses like <code class="language-plaintext highlighter-rouge">http://johnsmith.s3.amazonaws.com/photos/puppy.jpg</code>. Processing with a system that works in whole images, like ImageMagick, happens like this: <img src="/assets/images/magick-s3.png" alt="Processing with image-at-a-time systems" /> Reading from the left, first the data is downloaded from the bucket into a large area of memory, then the image is decompressed, then processed, perhaps in several stages, then recompressed, then finally uploaded back to cloud storage. Each stage must complete before the next stage can start, and each stage needs at least two large areas of memory to function. Current libvips is able to execute decode, process and encode all at the same time, in parallel, and without needing any intermediate images. It looks more like this: <img src="/assets/images/old-libvips-s3.png" alt="Processing with current libvips" /> Because the middle sections are overlapped we get much better latency: the total time the whole process takes from start to finish is much lower. However, current libvips still needs the compressed input image to be read to memory before it can start, and can鈥檛 start to upload the result to cloud storage until it has finished compressing the whole output image. This is where true streaming comes in. libvips git master can now decode directly from a pipe and encode directly to a pipe. It looks more like this: <img src="/assets/images/new-libvips-s3.png" alt="Processing with libvips streams" /> Now everything overlaps, and latency should drop again. <h1 id="api">API</h1> Here鈥檚 how it looks in Python: <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source = pyvips.Source.new_from_descriptor(4132) image = pyvips.Image.new_from_source(source, "") if image.width > 1000: # big image! .. shrink on load image = pyvips.Image.new_from_source(source, "", shrink=2) image = image.invert() target = pyvips.Target.new_to_descriptor(2487) image.write_to_target(target) </code></pre></div></div> The neat part is that you can open the source twice, once to get the header and decide how to process it, and a second time with the parameters you want. Behind the scenes, the source is buffering bytes as they arrive from the input. If you reuse the source, it鈥檒l automatically rewind and reuse the buffered bytes until they run out. Once you switch from reading the header to processing pixels, the buffer is discarded and bytes from the source are fed directly into the decompressor. The mechanism that supports this is set of calls loaders can use on sources to hint what kind of access pattern they are likely to need, and what part of the image (header, pixels) they are working on. <h1 id="custom-sources">Custom sources</h1> libvips ships with streams that can attach to files, areas of memory, and file descriptors (eg. pipes). You can add your own connection types by subclassing <code class="language-plaintext highlighter-rouge">VipsSource</code> and <code class="language-plaintext highlighter-rouge">VipsTarget</code> and implementing <code class="language-plaintext highlighter-rouge">read</code> and <code class="language-plaintext highlighter-rouge">write</code> methods, but this can be awkward for languages other than C or C++. To make custom streams easy in languages like Python, there are classes called <code class="language-plaintext highlighter-rouge">VipsSourceCustom</code> and <code class="language-plaintext highlighter-rouge">VipsTargetCustom</code>. You can make your own stream objects like this: <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file = open(sys.argv[1], "rb") def read_handler(size): return file.read(size) source = pyvips.SourceCustom() source.on_read(read_handler) </code></pre></div></div> This makes a very simple source which just reads from a file. Without a seek handler, <code class="language-plaintext highlighter-rouge">Source</code> will treat this as a pipe and do automatic header buffering. Like any source, you can use it to make an image: <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>image = pyvips.Image.new_from_source(source, '') </code></pre></div></div> Or perhaps: <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>image = pyvips.Image.thumbnail_source(source, 128) </code></pre></div></div> You could make one with a seek handler like this: <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file = open(sys.argv[1], "rb") def read_handler(size): return file.read(size) def seek_handler(offset, whence): file.seek(offset, whence) return file.tell() source = pyvips.Source() source.on_read(read_handler) source.on_seek(seek_handler) </code></pre></div></div> A seek method is optional, but will help file formats like TIFF which seek a lot during read. <h1 id="custom-output-streams">Custom output streams</h1> Output streams are almost the same: <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file = open(sys.argv[2], "wb") def write_handler(chunk): return file.write(chunk) def finish_handler(): file.close() target = pyvips.TargetCustom() target.on_write(write_handler) target.on_finish(finish_handler) </code></pre></div></div> So you can now do this! <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>image = pyvips.Image.new_from_source(source, '') image.write_to_target(target, '.png') </code></pre></div></div> And it鈥檒l copy between your two objects. <h1 id="loader-and-saver-api">Loader and saver API</h1> There鈥檚 quite a large chunk of new API for loaders and savers to use to hook themselves up to streams. We鈥檝e rewritten jpg, png, webp, hdr (Radiance), tif (though only load, not save), svg and ppm/pfm/pnm to work only via this new class. We plan to rework more loaders and savers in the next few libvips versions. The old file and buffer API will become a thin layer over the new connection system. <footer class="site-footer"> <a href="https://github.com/libvips/libvips">View on GitHub</a> <a style='float: right;' href="/feed.xml"><img src="/assets/images/rss.png" alt="subscribe via rss"/></a> </footer> </section> <script type="text/javascript"> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-48550036-2', 'auto'); ga('send', 'pageview'); </script> </body> </html>